A training method and device of a multitask processing model, a storage medium and an electronic device

Abstract

The embodiment of the specification discloses a kind of training method, device, storage medium and electronic equipment of multi-task processing model, wherein, method includes: creating initial multi-task processing model, initialization shared basic mask score tensor and exclusive residual mask score tensor, based on shared basic mask score tensor and exclusive residual mask score tensor, each computer processing task is generated task effective weight to carry out forward propagation processing computer processing task prediction result determination computer processing task task loss to determine gradient information, based on gradient information and model connection weight parameter determination composite dynamic importance score and adjustment is carried out to shared basic mask score tensor and exclusive residual mask score tensor, based on task loss, shared basic mask score tensor, task exclusive residual mask score tensor and composite dynamic importance score determine model composite total loss carries out model parameter adjustment to get multi-task processing model.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a training method, apparatus, storage medium, and electronic device for a multi-task processing model. Background Technology

[0002] In modern computing service scenarios, it is often necessary to efficiently handle multiple related but different computer processing tasks using a single machine learning model. For example, in cloud-based image vision processing services, it is necessary to simultaneously perform computer vision processing tasks related to image analysis, such as face recognition, object detection, and scene segmentation. To address this, Multi-Task Learning (MTL) technology has emerged. MTL improves the generalization ability and learning efficiency of a model by sharing network parameters across different tasks and utilizing complementary information between tasks.

[0003] However, traditional "hard parameter sharing" generally faces two major challenges: First, when there are conflicts between different types of processing tasks, forced parameter sharing is prone to "negative migration" and affects service accuracy; Second, in order to accommodate diverse task knowledge, the sharing network is usually designed to be very large, which brings great difficulties to the real-time response of the model and its deployment on edge nodes or terminal devices. Summary of the Invention

[0004] This specification provides a training method, apparatus, storage medium, and electronic device for a multi-task processing model, the technical solution of which is as follows: Firstly, embodiments of this specification provide a training method for a multi-task processing model, characterized by comprising the following steps: Identify multiple computer processing tasks of different task types in a computer service scenario, create an initial multi-task processing model for the computer service scenario, initialize a shared basic mask score tensor shared by all the computer processing tasks for the model connection weight parameters of the initial multi-task processing model, and initialize a dedicated residual mask score tensor for each computer processing task. During model training, effective weights are generated for each computer processing task based on the shared base mask score tensor and the specific residual mask score tensor of each computer processing task. Forward propagation is performed on each computer processing task based on the effective weights to obtain the prediction result of the computer processing task. The task loss of each computer processing task is determined based on the prediction result of the computer processing task. Based on the determined gradient information of each of the aforementioned computer processing tasks, and based on the gradient information and the model connection weight parameters, a composite dynamic importance score for each connection weight in the model connection weight parameters for each of the aforementioned computer processing tasks is determined; Based on the composite dynamic importance score, the model connection weight mask is adjusted for the shared basic mask score tensor and the dedicated residual mask score tensor of each computer processing task; The model's composite total loss is determined based on the task loss of each computer processing task, the shared basic mask score tensor, the task-specific residual mask score tensor, and the composite dynamic importance score. Based on the model's composite total loss, the model connection weight parameters, the shared basic mask score, and the task-specific residual mask score are adjusted until the model training ends and a trained multi-task processing model is obtained.

[0005] Secondly, embodiments of this specification provide a training apparatus for a multi-task processing model, the apparatus comprising: The model creation module is used to determine multiple computer processing tasks of different task types in a computer service scenario, create an initial multi-task processing model for the computer service scenario, initialize the model connection weight parameters of the initial multi-task processing model, initialize a shared basic mask score tensor shared by all the computer processing tasks, and initialize a dedicated residual mask score tensor for each computer processing task. The model training module is used to generate effective weights for each computer processing task based on the shared base mask score tensor and the specific residual mask score tensor of each computer processing task during the model training process, perform forward propagation processing on each computer processing task based on the effective weights to obtain the prediction result of the computer processing task, and determine the task loss of each computer processing task based on the prediction result of the computer processing task. The model training module is used to determine the composite dynamic importance score of each connection weight in the model connection weight parameters for each of the computer processing tasks based on the determined gradient information of each of the computer processing tasks and based on the gradient information and the model connection weight parameters. The model training module is used to adjust the model connection weight mask based on the composite dynamic importance score for the shared base mask score tensor and the dedicated residual mask score tensor of each computer processing task; The model's composite total loss is determined based on the task loss of each computer processing task, the shared basic mask score tensor, the task-specific residual mask score tensor, and the composite dynamic importance score. Based on the model's composite total loss, the model connection weight parameters, the shared basic mask score, and the task-specific residual mask score are adjusted until the model training ends and a trained multi-task processing model is obtained.

[0006] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0007] Fourthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0008] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following: In one or more embodiments of this specification, the electronic device creates an initial multi-task processing model, initializes a shared base mask score tensor and a dedicated residual mask score tensor, generates effective task weights for each computer processing task based on the shared base mask score tensor and the dedicated residual mask score tensor, performs forward propagation processing on each computer processing task, determines the task loss of each computer processing task based on the prediction results, determines gradient information, determines a composite dynamic importance score based on the gradient information and model connection weight parameters, and adjusts the shared base mask score tensor and the dedicated residual mask score tensor, and determines the model's composite total loss based on the task loss, the shared base mask score tensor, the task-specific residual mask score tensor, and the composite dynamic importance score, and adjusts the model parameters to form a multi-task processing model; through a hierarchical sparse mask structure containing one shared base mask and multiple task-specific residual masks, a dedicated sub-network containing both shared and independent knowledge is constructed for each computer processing task. This approach fundamentally solves the "negative transfer" phenomenon caused by forced parameter sharing, effectively isolates gradient conflicts between different computer processing tasks, and thus significantly improves the processing performance and accuracy of the multi-task processing model on each task. Furthermore, highly sparse multi-task processing models can be trained through the above method, which greatly reduces the number of model parameters, storage requirements, and inference computation complexity. This effectively avoids models that are too large and difficult to deploy on resource-constrained terminals or cost-sensitive computer service scenarios, achieving a synergistic improvement in model performance and operating efficiency. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating a training method for a multi-task processing model provided in the embodiments of this specification. Figure 2 This is a schematic diagram of a forward propagation process provided in an embodiment of this specification; Figure 3 This is a schematic diagram of a process for determining a composite dynamic importance score provided in the embodiments of this specification; Figure 4 This is a schematic diagram of a model connection weight mask adjustment process provided in the embodiments of this specification; Figure 5 This is a schematic diagram of the structure of a training device for a multi-task processing model provided in the embodiments of this specification; Figure 6 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0011] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0012] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0013] In related technologies, the traditional "hard parameter sharing" (MTL) method faces a severe challenge: negative transfer. When there are significant differences or conflicts between tasks, the gradient update directions of the shared network may contradict each other in order to accommodate all tasks, resulting in final performance that is even worse than models trained separately for each task. In addition, in order to accommodate knowledge from multiple tasks, the shared backbone network is usually designed to be very large, which is unacceptable in deployment environments with limited computing resources, power consumption, and memory (such as mobile devices, autonomous vehicles, and IoT terminals). To solve the problem of large models, network sparsification or pruning techniques have emerged. The core idea is to remove redundant or unimportant weight connections in the network to form a "sparse" subnetwork, thereby significantly reducing the number of parameters, storage space, and computational cost (FLOPs) during inference without significantly affecting (and sometimes even improving) the model performance.

[0014] Sparsity reduction methods in related technologies can be mainly divided into: 1. Static Sparsity: The sparse structure is determined once after training or during training and remains unchanged afterward. For example, the classic lottery hypothesis suggests that in a large, dense network, there exists a "winning lottery" subnetwork that, when trained alone, can achieve performance comparable to the original network. However, finding a general "winning lottery" for multiple tasks also faces the problem of task conflict; a connection crucial for task A may be redundant for task B.

[0015] 2. Dynamic Sparse Training (DST): During training, the sparse structure is periodically adjusted, essentially "pruning" currently unimportant connections and "regenerating" new, potentially more important connections. These methods (such as RigL and SET) allow the sparse structure to dynamically adapt to changes in data and tasks, typically achieving better performance than static sparse training. However, most existing DST methods are designed for single-task scenarios. When directly applied to MTL, they usually maintain a single sparse mask for all tasks, failing to fundamentally resolve conflicts between tasks.

[0016] In summary, a significant gap exists in modern computer service scenarios: a lack of a systematic approach that combines the flexibility of dynamic sparsity with the specific needs of multi-task learning (sharing knowledge while isolating conflicts). How to enable multi-task processing models to dynamically "sculpt" optimal network structures that possess both shared and dedicated components for different tasks is a pressing technical challenge that needs to be addressed in this field.

[0017] The present specification will now be described in detail with reference to specific embodiments.

[0018] In one embodiment, such as Figure 1 As shown, a training method for a multi-task processing model is proposed. This method can be implemented using a computer program and can run on a training device for a multi-task processing model based on the von Neumann architecture. This computer program can be integrated into applications or run as a standalone utility application. The training device for the multi-task processing model can be an electronic device, including but not limited to: personal computers, tablets, handheld devices, in-vehicle devices, wearable devices, computing devices, or other processing devices connected to a wireless modem. Terminal devices can have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user equipment, cellular phone, cordless phone, device in 5G network or future evolved network, etc.

[0019] Specifically, the training methods for this multi-task processing model include: S102: Determine multiple computer processing tasks of different task types under a computer service scenario, create an initial multi-task processing model for the computer service scenario, initialize a shared basic mask score tensor shared by all the computer processing tasks for the model connection weight parameters of the initial multi-task processing model, and initialize a dedicated residual mask score tensor for each computer processing task. Computer service scenarios refer to application environments that provide multi-task processing services through electronic devices. For example, in intelligent driving perception service scenarios, multi-task processing models can be applied to an in-vehicle computing platform to simultaneously perform visual perception tasks of different types, such as object detection, drivable area segmentation, and lane detection. Another example is medical image analysis service scenarios, where multi-task processing models can be applied to a medical AI-assisted diagnostic platform to process 3D images from different modalities (such as CT and MRI) and simultaneously perform visual perception tasks of different types, such as lung nodule detection (based on CT) and brain tumor segmentation (based on MRI). Initial multi-task processing model: The multi-task processing model during the model training phase.

[0020] In one embodiment, the initial multi-task processing model is a pre-defined multi-task neural network created based on a machine learning model (its dense weight parameters are denoted as...). (where N is the total number of parameters), construct a learnable, hierarchical sparse structure parameterization system.

[0021] In this specification, dense weight parameters refer to the model connection weight parameters of the multi-task processing model. They can be understood as the weight values ​​of the connections between neurons that constitute the initial multi-task processing model. They are the core parameters that determine the model's function and can also be simply referred to as weights or connection weights.

[0022] Optionally, the initial multitasking model may include a backbone network shared by all computers processing tasks, and a task head network designed for each computer processing task and independent of other tasks. In the initialization phase, this model is typically a standard dense model.

[0023] In practical applications, for the initial multi-task processing model, the two classes are initialized with dense weight parameters similar to those of the initial multi-task processing model. Mask score tensors of the same dimension can be optimized during model training using standard gradient descent. Mask score tensors are categorized as follows: Shared Base Mask Score Tensor (S0): This is a score tensor shared by all tasks. Each element represents a shared base mask score. Each of its elements Corresponding weight Importance in a shared underlying network. This tensor is shared by all computer processing tasks, and each score value is intended to characterize the general importance or shared value of the corresponding connection weight for all tasks. Optionally, the tensor is initialized with a normal distribution with a mean of 0 and a standard deviation of 0.01.

[0024] Dedicated residual mask scoring tensor (for (K tasks): Pre-initialize a dedicated connection weight parameter with the model for each of the K tasks k. A score tensor of the same dimension, each element Corresponding weight The importance of the tensors in the dedicated network for task k. Optionally, this set of tensors can be initialized using a normal distribution with a mean of 0 and a standard deviation of 0.01.

[0025] Illustratively, first, the applicable computer service scenario is determined, and within this scenario, multiple different types of computer processing tasks need to be handled by a single model. Different task types imply differences in data modality, processing logic, or optimization objectives. Second, based on the determined computer service scenario and task types, a suitable initial multi-task processing model is created or selected. This initial multi-task processing model serves as a foundational architecture, with its connection weights defined by the model connection weight parameters. For example, a general pre-trained model (such as BERT, ResNet, Transformer, etc.) can be used as a shared backbone network, upon which multiple parallel task heads are built to correspond to different computer processing tasks. For all or part of the model connection weight parameters of the created initial multi-task processing model, two types of learnable score tensors are initialized, which together form the basis of the subsequent dynamically evolving hierarchical structure: Shared base mask score tensor: Initialize a tensor shared by all computers processing tasks. This tensor corresponds one-to-one with the model connection weights, providing the foundation for learning a "shared subnetwork" that is crucial to all tasks.

[0026] Dedicated Residual Mask Score Tensor: This process iterates through all computer processing tasks and initializes a task-specific residual mask score tensor for each task. If there are K tasks, K tensors of this type will be initialized. Each tensor also corresponds one-to-one with the model connection weights, providing the foundation for learning the unique "supplementary subnetworks" for each task.

[0027] In an alternative implementation, all mask score tensors can be initialized with a specific random distribution (e.g., a normal distribution with a mean of 0 and a standard deviation of 0.01) to ensure the randomness and fairness of the initial state.

[0028] Example: Scenario: Cloud-based content security auditing service of a large e-commerce platform.

[0029] Computer processing tasks: This service requires real-time review of product information (including images and text) uploaded by users, which includes three different types of processing tasks: violation image recognition (an image classification task), advertising text detection (a sequence labeling task in natural language processing), and product category identification (a multimodal classification task).

[0030] Execute S102: First, the e-commerce content review service scenario and three specific computer processing tasks were identified.

[0031] Next, a pre-trained multimodal model (e.g., ViLT - Vision-and-LanguageTransformer) was selected as the shared backbone network of the initial multi-task processing model, and three different task heads were built on it, corresponding to the three tasks mentioned above.

[0032] Finally, the system performed initialization operations on the model connection weight parameters of the model: a shared base mask score tensor shared by all review tasks was initialized for the three tasks of "illegal image recognition", "advertising text detection" and "product category identification", and three independent dedicated residual mask score tensors were initialized.

[0033] S104: During model training, effective weights are generated for each computer processing task based on the shared base mask score tensor and the dedicated residual mask score tensor of each computer processing task. Forward propagation is performed on each computer processing task based on the effective weights to obtain the prediction result of the computer processing task. The task loss of each computer processing task is determined based on the prediction result of the computer processing task. Task-effective weights refer to a set of weight parameters dynamically generated for the current model calculation in each forward propagation of the model, specifically for the corresponding computer processing task. Task-effective weights consist of the full model connection weight parameters of the model and a valid mask generated for the task model, and are used only for the current forward propagation calculation of that specific task.

[0034] Forward Propagation Process: This refers to the process of using the input data of a corresponding computer processing task through an (initial) multi-task processing model to generate a prediction result. In this specification, the forward propagation process is characterized by the fact that the neural network structure it relies on is dynamically defined by the "task effective weights".

[0035] Computer-aided task prediction results refer to the output of the model after performing forward propagation processing on the input data. Its specific form depends on the task type; for example, for classification tasks, the prediction result can be the class probability; for detection tasks, it can be bounding box coordinates and class information.

[0036] Task loss: This refers to a parameter used to quantify the difference between the "computer processing task prediction" and the true label for a given computer processing task. The smaller this parameter, the more accurate the model's prediction.

[0037] To illustrate, for any computer processing task k to be processed in the current iteration, it is first necessary to synthesize a dedicated and effective mask for it. This process takes the two types of mask score tensors initialized in the previous step as input: namely, the shared basic mask score tensor S0 and the task-specific residual mask score tensor. In an alternative implementation, the system can first transform the two types of score tensors into probabilistic relaxed masks using a differentiable transformation function (such as Gumbel-Softmax), and then combine them into a final task-specific effective mask using a merging operation (such as logical union). .

[0038] After obtaining the task-specific effective mask, the system applies it to the full model connection weight parameters θ to generate the task-specific effective weights. In one embodiment, this application process can be achieved by element-wise multiplying the effective mask with the model connection weight parameters. The resulting task-effective weights essentially define a sparse subnetwork structure tailored to task k.

[0039] Next, the effective weights of the tasks generated in the previous step are used to configure the initial multi-task processing model. Then, the input data corresponding to task k is fed into this dynamically configured initial multi-task processing model for network parameterization. Then, a complete forward propagation process is performed, that is, the (sample) input data of the corresponding computer processing task is fed into the initial multi-task processing model. The output of the processing is the prediction result of the computer processing task.

[0040] Finally, the computer processing task prediction results obtained from the forward propagation are compared with the preset true labels. For example, a scalar value can be calculated using a loss function suitable for the corresponding computer processing task type (e.g., the cross-entropy loss function for classification tasks, or the mean squared error loss function for regression tasks). This value is the task loss L for the computer processing task k. k .

[0041] It should be noted that for a training batch containing multiple different computer processing tasks, the above processing procedure will be executed independently, ultimately resulting in a set of task loss vectors corresponding to each task. .

[0042] Example: Taking the content security review service of the e-commerce platform in the above example as an example: In one training iteration, the system receives a mixed batch of data, which includes a set of images for the "violation image recognition" task and a set of product titles for the "advertising text detection" task.

[0043] Execute S104: For the "violation image recognition" task: We extract the shared basic mask score tensor S0 and the task-specific residual mask score tensor ΔS (image). Based on these two, we generate an effective weight for image recognition. Using this effective weight configuration, we perform forward propagation on the input image data to obtain the probability of each image being a violation, which is then used as the prediction result. We compare the predicted probability with the true violation label and calculate the task loss L (image) using the cross-entropy loss function.

[0044] For the "advertising text detection" task: We extract the shared base mask score tensor S0 and the task-specific residual mask score tensor ΔS(text). Based on these two, we generate a task-effective weight that is completely different from that used in image tasks, specifically for text detection. Using this effective weight configuration, we perform forward propagation on the input product title to obtain a sequence of labels indicating whether each character is an advertising phrase, which serves as the prediction result.

[0045] The predicted label sequence is compared with the actual ad label sequence, and the task loss L(text) is calculated using a sequence labeling loss function (such as CRF loss).

[0046] S106: Based on the determined gradient information of each of the computer processing tasks, and based on the gradient information and the model connection weight parameters, determine the composite dynamic importance score of each connection weight in the model connection weight parameters for each of the computer processing tasks; Gradient information refers to the partial derivative of the task loss with respect to the model's connection weights. Gradient information is typically calculated using the backpropagation algorithm. It not only indicates the direction in which model parameters need to be adjusted to reduce loss, but its magnitude also reflects the sensitivity or influence of specific parameters on the current task loss.

[0047] Composite dynamic importance score: refers to a numerical measure used to dynamically and comprehensively evaluate the contribution of each model connection weight to a specific computer processing task.

[0048] This is illustrative of the forward propagation process after completing all tasks and calculating the respective losses. Then, backpropagation is performed, and through the automatic differentiation engine, the loss for each task can be efficiently calculated with respect to the model connection weight parameters. gradient These gradient information along with the weights This itself forms the basis for assessing the importance of each connection.

[0049] After obtaining the gradient information, the system combines this gradient information with the model connection weight parameters themselves to calculate an independent composite dynamic importance score for each connection weight i in the model, for each computer processing task k. For a model with N weights and K tasks, this step will calculate N * K importance scores.

[0050] In an optional implementation, to make the assessment more comprehensive and accurate, the composite dynamic importance score can be obtained by weighted summation of at least one or more of the following sub-indicators: Weight magnitude: that is, the absolute value of the connection weight parameters in the model. , representing the static strength of the connection.

[0051] Gradient significance: can be based on gradient information and weight values The calculated terms (e.g., | The symbol | represents the immediate impact of the connection on the current task's loss.

[0052] Structural stability: can be based on historical values ​​of gradient information (e.g., exponential moving average of the squared gradient). The value is approximated by ), representing the long-term impact of this connection on the model's output structure.

[0053] S108: Based on the composite dynamic importance score, perform model connection weight mask adjustment on the shared basic mask score tensor and the dedicated residual mask score tensor of each computer processing task; Model connection weight mask adjustment refers to the process of modifying the values ​​in the shared base mask score tensor and / or the dedicated residual mask score tensor. The purpose of this adjustment is not to directly change the model connection weight parameters themselves, but rather to change the form of the "mask" used to generate effective weights for the task, thereby indirectly altering and optimizing the sparse topology of the multi-task processing model.

[0054] The methods for adjusting the model connection weight mask can be pruning operations and regrowth operations. Pruning operation: refers to changing some of the connections that are currently in an "active" state to an "inactive" state.

[0055] Regeneration operation: refers to transforming a portion of connections that are currently in an "inactive" state into an "active" state.

[0056] Optionally, step S108 is not executed in every training iteration, but can be triggered periodically according to a preset update cycle.

[0057] This illustrates how differentiated adjustment strategies are employed for different types of mask score tensors to achieve synergistic optimization of globally shared knowledge and locally specific knowledge.

[0058] Adjustments to the dedicated residual mask score tensor: When adjusting the residual mask score tensor for a specific computer processing task k, the system uses only the composite dynamic importance score corresponding to that task as the basis for ranking and decision-making. In one embodiment, this adjustment can employ a "pruning-regeneration" mechanism: a. Pruning: Among the connections currently active for task k, identify the batch of connections with the lowest importance scores and lower their corresponding mask scores.

[0059] b. Regeneration: Among the connections that are not currently active for task k, identify the batch of connections with the highest importance scores and increase their corresponding mask scores.

[0060] Adjustments to the shared underlying mask score tensor: When adjusting the shared underlying mask score tensor shared by all tasks, a metric reflecting global consensus is needed because it affects all tasks. Therefore, before adjustment, the composite dynamic importance scores of all tasks are first weighted and aggregated to generate a global importance score. Subsequently, the system uses this aggregated global importance score as the basis to perform the same "pruning-regeneration" adjustment on the shared underlying mask score tensor as described above.

[0061] Through this layered and differentiated adjustment mechanism, the model can flexibly and quickly optimize the specific knowledge modules for each task while maintaining the stability of the shared knowledge base.

[0062] S110: Determine the model composite total loss based on the task loss of each computer processing task, the shared basic mask score tensor, the task-specific residual mask score tensor, and the composite dynamic importance score. Adjust the model parameters, the shared basic mask score, and the task-specific residual mask score based on the model composite total loss until the model training ends and a trained multi-task processing model is obtained.

[0063] Model composite total loss: refers to a comprehensive scalar loss value used to guide the overall optimization of the model. It does not only measure the model's prediction accuracy on the task, but is a composite function that integrates multiple optimization objectives (e.g., task performance, model sparsity, structural rationality, etc.).

[0064] Model parameter tuning: During machine learning training, based on the gradient information calculated by the loss function, an optimization algorithm (such as stochastic gradient descent SGD, Adam, etc.) is used to update the values ​​of all learnable parameters in the model in order to obtain a lower loss value in the next iteration.

[0065] Illustratively, a model composite total loss (L_total) is defined, for example, this total loss is constructed based on multiple information sources, including: The task loss for each computer processing task, the shared basic mask score tensor, and the task-specific residual mask score tensor.

[0066] In one specific implementation, the total loss of the model can be composed of a weighted combination of the following three parts: a. Task Loss Item (L_task): This item is directly based on the task loss (L_task) incurred by each computer in processing the task. k This is calculated, for example, by weighted summation of the losses across all tasks. This term is the main driving force behind the model's improved task performance.

[0067] b. Global Sparsity Regularization Term (L_sparsity): This term is mainly calculated based on the shared base mask score tensor and the task-specific residual mask score tensor. Its function is to introduce a "soft constraint" on the overall sparsity of the model, guiding the model to evolve towards the preset sparsity target.

[0068] c. Structural coherence regularization term (L_coherence): This term is mainly calculated based on the task-specific residual mask score tensor. Its function is to penalize the activation of task-specific connections, thereby encouraging the model to prioritize the reuse of connections defined by the shared basic mask and promoting knowledge sharing.

[0069] Then, a unified model parameter tuning is performed. After determining the model's total composite loss (L_total), the gradient of this total loss with respect to all learnable parameters in the model can be calculated using an automatic differentiation framework. Subsequently, an optimizer (such as AdamW) is used to perform a model parameter tuning on all learnable parameters using this gradient information. The parameter objects that are tuned and updated include: model connection weight parameters, shared base mask score tensor, and task-specific residual mask score tensor.

[0070] The above approach ensures that the model’s functionality (determined by the weights) and its sparse structure (determined by the mask score) can be optimized collaboratively and end-to-end.

[0071] Then iterate through the training until completion. The complete process from S104 to S110 constitutes a single training iteration. This iterative process is repeated until one or more preset training termination conditions are met (e.g., reaching the maximum number of iterations, model performance converging on the validation set, etc.). After training, the final result is the trained multi-task processing model.

[0072] In the embodiments of this specification, the electronic device creates an initial multi-task processing model, initializes a shared base mask score tensor and a dedicated residual mask score tensor, generates effective task weights for each computer processing task based on the shared base mask score tensor and the dedicated residual mask score tensor, performs forward propagation processing on each computer processing task, determines the task loss of each computer processing task based on the prediction results, determines gradient information, determines a composite dynamic importance score based on the gradient information and model connection weight parameters, and adjusts the shared base mask score tensor and the dedicated residual mask score tensor, and determines the model's composite total loss based on the task loss, the shared base mask score tensor, the task-specific residual mask score tensor, and the composite dynamic importance score, and adjusts the model parameters to form a multi-task processing model; through a hierarchical sparse mask structure containing one shared base mask and multiple task-specific residual masks, a dedicated sub-network containing both shared and independent knowledge is constructed for each computer processing task. This approach fundamentally solves the "negative transfer" phenomenon caused by forced parameter sharing, effectively isolates gradient conflicts between different computer processing tasks, and thus significantly improves the processing performance and accuracy of the multi-task processing model on each task. Furthermore, highly sparse multi-task processing models can be trained through the above method, which greatly reduces the number of model parameters, storage requirements, and inference computation complexity. This effectively avoids models that are too large and difficult to deploy on resource-constrained terminals or cost-sensitive computer service scenarios, achieving a synergistic improvement in model performance and operating efficiency.

[0073] Optional, please see Figure 2 , Figure 2 This is a schematic diagram of a forward propagation process proposed in this specification. Specifically, it involves generating effective task weights for each computer processing task based on the shared base mask score tensor and the specific residual mask score tensor of each computer processing task, and performing forward propagation processing on each computer processing task based on the effective task weights to obtain the prediction result of the computer processing task, including: S202: Based on the shared basic mask score tensor and the dedicated residual mask score tensor of each computer processing task, a connection weight graph is generated for each computer processing task to obtain the task-specific relaxed residual mask and the task relaxed basic mask. Connection weight graph generation: This refers to the process of transforming a learnable, arbitrary real-valued mask "score" tensor (shared base mask score tensor and task-specific residual mask score tensor) into a "mask" tensor (typically between 0 and 1) that can be directly used for computation and represents the activation probability of connection weights in the model. The result is a task-specific relaxed residual mask and a task-relaxed base mask. Task-relaxed base mask: refers to the probability mask obtained after the shared base mask score tensor has undergone the "compute connection weight graph generation" step. Here, "relaxed" means that its element values ​​are continuous values ​​between 0 and 1, rather than strictly binary 0 or 1. This characteristic is key to ensuring that the entire training process can be optimized through gradient descent.

[0074] Task-specific relaxation residual mask: A probability mask with the same properties as the aforementioned relaxation basic mask, obtained by processing the task-specific residual mask score tensor through the "calculation of connection weight graph generation" step. To illustrate, for the current computer processing task k, two input tensors are first determined: a globally shared basic mask score tensor and a task-specific residual mask score tensor. The "compute connection weight graph generation" operation is then performed independently on each of the two score tensors.

[0075] In an alternative implementation, this operation can be achieved using a differentiable function capable of generating approximately discrete samples from a continuous distribution, such as the Gumbel-Softmax function. This function maps each score value in the score tensor to a probability value between 0 and 1, which can be interpreted as the degree to which the corresponding connection weights are activated.

[0076] Then, the output consists of two probability mask tensors with the same dimension as the input score tensor (the same score tensor dimension as the shared base mask score tensor and the dedicated residual mask score tensor for each of the computer processing tasks): a task relaxation base mask generated from the shared base mask score tensor, and a task-specific relaxation residual mask generated from the dedicated residual mask score tensor corresponding to task k.

[0077] In one specific implementation, at the beginning of the forward propagation of each training iteration, these static score tensors are dynamically transformed into relaxation masks for the current computation. Specifically, the process of calculating the connection weight graph for each computer processing task based on the shared base mask score tensor and the task-specific residual mask score tensor to obtain the task-specific relaxation residual mask and task-specific relaxation base mask can be performed as follows: Step A2: Sample noise tensors with the same mask tensor dimension from a preset random probability distribution; Sample a noise tensor of the same dimension as the score tensor from the standard Gumbel distribution. Each of its elements ,in . These are random values ​​randomly sampled from a uniform distribution of 0 and 1; Step A4: Based on the noise tensor and the preset temperature hyperparameter, perform masking calculation on the shared basic masking score tensor using the first calculation formula to obtain the task relaxation basic mask; The first calculation formula satisfies the following formula:

[0078] Among them, the For the i-th element of the task relaxation base mask, the For the i-th element of the shared basic mask score tensor, the For the i-th element of the noise tensor with respect to the shared underlying mask score tensor, the This refers to the preset temperature hyperparameter; Step A6: Based on the noise tensor and the preset temperature hyperparameter, perform a second mask calculation on the second formula of the dedicated residual mask score tensor of each computer processing task to obtain the task-specific relaxation residual mask. The second calculation formula satisfies the following formula:

[0079] Among them, the The i-th element of the task-specific relaxation residual mask for the k-th computer processing task, For the i-th element of the dedicated residual mask score tensor of the k-th computer processing task, the Let i be the i-th element of the noise tensor for the k-th computer processing task.

[0080] Schematic, steps A4-A6 are performed to calculate the relaxed samples, that is, the noise tensor is added to the mask score tensor, and then passed through a temperature-controlled... The Sigmoid function. Therefore, relax the underlying mask. and K relaxed residual masks Each element is calculated with reference to the first and second calculation formulas.

[0081] Furthermore, It's the Sigmoid function. Temperature parameter. It is a key hyperparameter, and the model training process employs a cosine annealing strategy to initialize it from a relatively high initial value (e.g., ...). Decays exponentially or cosinely to a lower final value (e.g.) This allows the model to explore structures extensively in the early stages of training (with mask values ​​close to 0.5), while in the later stages of training it can make more deterministic structure choices (with mask values ​​approaching 0 or 1).

[0082] S204: Based on the task-specific relaxation residual mask and the task relaxation base mask, a mask combination is performed on each of the computer processing tasks to obtain a task-specific computation mask. The task-specific computation mask is applied to the model connection weight parameters to obtain the effective weight of each of the computer processing tasks. The initial multi-task processing model is then controlled to perform forward propagation processing on each of the computer processing tasks based on the effective weight of the tasks to obtain the prediction result of the computer processing tasks.

[0083] Mask combination refers to the process of integrating the task relaxation base mask and the corresponding task-specific relaxation residual mask for computer processing tasks to generate a model connection weight mask for computer processing tasks. The result of the combination is a task-specific computational mask. Task effective weights: refer to the weight parameters used for the current forward propagation process after applying the task-specific computation mask to the full model connection weight parameters.

[0084] Indicative, for a specific task The final effective relaxation mask used in its forward propagation The mask is synthesized from the task-specific relaxation base mask and its task-specific relaxation residual mask through a logical union operation. Under the setting of continuous relaxation, this is equivalent to the union calculation of probabilities, and the mask combination can be referred to the following formula:

[0085] in, This represents the Hadamard product (element-wise multiplication). This synthesized task-specific computational mask. The task was defined. In a specific computation graph in the current iteration, the value of each element (between 0 and 1) can be seen as the degree to which the corresponding weight is "activated". The relaxed base mask, also known as the "base mask" shared by all tasks, represents general knowledge. The K relaxed residual masks, also known as task-specific "residual masks", represent the unique knowledge of that task.

[0086] Then, we synthesize the mask. Dense weights applied to the original initial multi-task processing model Received the task The effective weights of the task in this forward propagation can be expressed as:

[0087] Use this set as a task Dynamically generated effective weights are used to perform a complete network forward propagation to obtain the prediction results for the computer processing task. .

[0088] In the embodiments of this specification, S202 is performed to transform the learnable mask score into a differentiable relaxed mask, and S204 is performed to further combine and generate a task-specific computational mask, thereby dynamically constructing a dedicated, sparse computational path for each computer processing task in each training iteration. This ability to tailor computational paths for different tasks is the physical basis for achieving shared knowledge and isolating task conflicts, providing a direct and effective technical means to ultimately alleviate the "negative transfer" phenomenon in multi-task learning.

[0089] Meanwhile, because the generated task weights are sparse, this process significantly reduces the actual computational cost during forward propagation, improving training efficiency. More importantly, the entire dynamic generation process maintains end-to-end differentiability, enabling the model not only to learn weight parameters but also to automatically learn and optimize its own sparse topology through gradient descent, achieving a high degree of synergy between function and structure.

[0090] Optional, please see Figure 3 , Figure 3 This is a flowchart illustrating a composite dynamic importance score determination method proposed in this specification. Specifically, the process involves determining gradient information based on each of the aforementioned computer processing tasks, and then, based on this gradient information and the model connection weight parameters, determining the composite dynamic importance score of each connection weight in the model connection weight parameters for each of the aforementioned computer processing tasks. This can be achieved using the following method: S302: Determine gradient information for the connection weight parameters of the model based on the task loss of each of the computer processing tasks; For each task loss, backpropagation is performed to calculate the partial derivative of that task loss with respect to each model connection weight parameter. The output of this process is the gradient information, which is in the form of the task loss with respect to the dense weights of the model. gradient .

[0091] S304: Based on the gradient information and the model connection weight parameters, determine the weight magnitude score, gradient significance score, and structural stability score of each connection weight in the model connection weight parameters for each computer processing task; The weight magnitude score satisfies the third calculation formula, the gradient significance score satisfies the fourth calculation formula, and the structural stability score satisfies the fifth calculation formula. The third calculation formula satisfies the following formula:

[0092] Among them, the Here, Ai is the i-th model connection weight among the model connection weight parameters, and Ai is the weight score of the i-th model connection weight among the model connection weight parameters. This indicates how to scale the values ​​in each dimension to... Normalization operation within an interval; The fourth calculation formula satisfies the following formula:

[0093] Wherein, Bi is the gradient significance score of the i-th model connection weight parameter, and L... k To represent the task loss of each of the aforementioned computer processing tasks; The fifth calculation formula satisfies the following formula:

[0094] Wherein, Ci is the structural stability score of the i-th model connection weight in the model connection weight parameters, and the... The output distribution influence parameter of the model for the connection weights of the i-th model;

[0095] The The model outputs the distribution influence parameter corresponding to the current training step t. It is the attenuation factor; S306: Based on the weight magnitude score, gradient significance score, and structural stability score, determine the composite dynamic importance score of each connection weight in the model connection weight parameters for each of the computer processing tasks.

[0096] Optionally, the composite dynamic importance score satisfies a sixth calculation formula, which in turn satisfies the following formula:

[0097] Among them, the For the composite dynamic importance score, These are the preset hyperparameters and satisfy... .

[0098] Schematic, for each weight in the model For each task Calculate a composite dynamic importance score This rating integrates three complementary dimensions, combined with the composite dynamic importance rating calculation formula of S304-S306, as follows:

[0099] in, These are preset hyperparameters (that satisfy) ), used to balance the contributions of the three dimensions. It is a hierarchical min-max normalization operator that scales the values ​​in each dimension to... Within the interval, to eliminate scale differences between different levels or indicators, ensuring the fairness of the weighted summation. First item This refers to the weight magnitude, representing the static strength of the weight. It's a classic pruning heuristic metric (its function is to measure the importance of a weight; intuitively, the larger the absolute value of the weight, the more important its role in the network). The second item... This is gradient significance, which is approximately equivalent to removing weights. (i.e., setting it to zero) for the loss function The first-order Taylor expansion of the resulting instantaneous changes can capture those connections that are crucial to the current optimization direction. The third term is structural stability, obtained through empirical Fisher information. This measures the influence of a parameter on the overall output distribution of the model. Directly calculating the Fisher information matrix is ​​costly, so we use a practical approximation of its diagonal, namely the square of the gradient. To make the estimate more stable and reflect the average effect over a period of time, it is smoothed and updated using an exponential moving average:

[0100] in This represents the current number of training steps. It is a decay factor close to 1 (e.g., 0.99). Using this formula, we assign each weight in the network a task-specific, dynamically changing, and richly detailed importance metric. This score considers not only the weight's own strength but also its immediate impact on the current task's optimization direction and its long-term impact on the model's output structure. This provides solid and reliable data support for subsequent structural adjustment steps such as "pruning" and "regeneration."

[0101] This specification establishes a dynamic evaluation framework comprising multiple complementary dimensions (i.e., static weight strength, immediate gradient impact, and long-term structural stability) using the aforementioned approach. This composite scoring mechanism provides a deeper and more accurate insight into the true contribution of each connection to a specific task, thereby offering a high-quality decision-making basis for subsequent network structure adjustments. This ensures that the model sparsification process is both efficient and accurate, maximizing the preservation of connections crucial to task performance.

[0102] Optional, please see Figure 4 , Figure 4 This is a flowchart illustrating a model connection weight mask adjustment process proposed in this specification. Specifically, the adjustment of the model connection weight mask based on the composite dynamic importance score for the shared base mask score tensor and the specific residual mask score tensor for each computer processing task can be performed as follows: S402: Determine the target mask score tensor corresponding to the current training step, and determine the number of pruned connections and the number of regrowth connections for the target mask score tensor; Target mask score tensor: This refers to the mask score tensor that is determined to need adjustment at the current training step, based on a preset update period. It can be a shared base mask score tensor or a task-specific residual mask score tensor.

[0103] Pruned connection count / Regenerated connection count: This refers to the specific number of connections that need to be pruned (deactivated) and regenerated (enabled) during the mask adjustment operation. In one embodiment, this count can be determined based on the amount of change in a dynamic sparsity scheduler s(t).

[0104] Indicatively, check if the current training step count satisfies the preset update period of a certain mask score tensor (e.g., t is a multiple of TΔ, or t is a multiple of T0). If satisfied, the corresponding mask score tensor (a specific residual mask or a shared base mask) is determined as the target mask score tensor for this adjustment. Based on a dynamic sparsity scheduler s(t), refer to the seventh formula to calculate the number of pruned connections and regrowth connections that need to be adjusted in this update.

[0105] In one feasible implementation, the process of determining the target mask score tensor corresponding to the current training step, and determining the number of pruned connections and the number of regrown connections for the target mask score tensor, can be performed in the following manner: Step B2: Based on the preset base mask training period and the current training step, determine whether the target mask score tensor corresponding to the current training step is a shared base mask score tensor; based on the preset dedicated mask training period and the current training step, determine whether the target mask score tensor corresponding to the current training step is a dedicated residual mask score tensor. Preset base mask training period: refers to a pre-set, fixed interval of training iterations, used to specify when to trigger the adjustment of the shared base mask score tensor.

[0106] Preset dedicated mask training period: refers to a pre-defined, fixed interval of training iterations, used to specify when to trigger adjustments to the dedicated residual mask score tensor. In one embodiment, this period value may be less than or equal to the preset base mask training period.

[0107] Current training steps: refers to the number of iterations completed from the start to the current moment during model training, usually represented by an integer variable (e.g., t).

[0108] Target mask score tensor: refers to one or more mask score tensors that are determined to need adjustment at the current training step, based on whether the preset training cycle conditions are met.

[0109] The process of determining the update of the shared base mask: The system performs a conditional judgment based on a preset base mask training period and the current training step number. In one specific implementation, this judgment might involve checking whether the current training step number is divisible by the preset base mask training period. If the judgment result is yes, then it is determined that the target mask score tensor corresponding to the current training step number contains the shared base mask score tensor.

[0110] The process of updating and determining the dedicated residual mask: Based on the preset dedicated mask training period and the current training step number, another conditional judgment is performed. In a specific implementation, this judgment may be to check whether the current training step number is divisible by the preset dedicated mask training period. If the judgment result is yes, the system determines that the target mask score tensor corresponding to the current training step number contains the corresponding dedicated residual mask score tensor.

[0111] It should be understood that, at certain specific training steps, both of the above conditions can be met simultaneously. In this case, the target mask score tensor will include both a shared base mask score tensor and one or more dedicated residual mask score tensors, which will be adjusted sequentially or in parallel in subsequent steps. If neither condition is met, mask adjustment will not be performed at the current training step.

[0112] Step B4: If the current training step number is less than or equal to the total number of steps in the warm-up phase, the seventh calculation formula is used to determine the current weight connection sparsity and the current target pruning total number corresponding to the current weight connection sparsity. Based on the target pruning total number and the previous target pruning total number corresponding to the previous training step, the current net pruning number is obtained. Based on the current net pruning number and the preset handover number, the number of pruned connections for the target mask score tensor is configured as the sum of the current net pruning number and the preset handover number, and the number of regrowth connections for the target mask score tensor is configured as the preset handover number. The seventh calculation formula satisfies the following formula:

[0113] Among them, the The target sparsity is for the preheating stage. The total number of steps in the warm-up phase, t is the current training step number, and the... Connect the sparsity of the current weights; Total steps in the warm-up phase: refers to the number of training steps preset for the warm-up phase at the beginning of training.

[0114] The current target total number of pruning branches, the cumulative total number of connection weights pruned across all training steps; The total number of target prunings refers to the number of target prunings determined at the training step of the last mask update.

[0115] Current net pruning count: This refers to the net increase in pruning connection weights required in this update to reach the current target total pruning count. Its value is equal to the difference between the current target total pruning count and the previous target total pruning count.

[0116] Preset number of handovers: This refers to a fixed number of handovers that are used to explore new connections in each adjustment. This parameter ensures that the network structure remains dynamically exploratory even as sparsity increases.

[0117] Indicatively, the seventh formula is used to determine the current target sparsity. Based on the current training step number t, the total number of warm-up steps T_warmup, and the final target sparsity S_target, the current weight connection sparsity s(t) is calculated.

[0118] Multiply s(t) by the total number of connections N in the model to obtain the total number of pruning branches for the current target.

[0119] Calculate the current net pruning count: Subtract the previous target pruning count corresponding to the previous training step from the current target total pruning count to obtain the current net pruning count (ΔN).

[0120] Finally, based on the calculated current net pruning number ΔN and the preset turnover number N_turnover, the system performs the final configuration: the number of regrowth connections is configured to the preset turnover number, and the number of pruning connections is configured to the sum of the current net pruning number and the preset turnover number.

[0121] Example: Using the content security audit service of an e-commerce platform as an example, and setting the following parameters: total number of model connections N=1,000,000, T_warmup=1000, S_target=0.8, update cycle TΔ=100, N_turnover=1,000.

[0122] Scenario: A mask adjustment was triggered during the 200th training iteration (t=200).

[0123] The execution process of B4: Condition judgment: t=200 is less than T_warmup=1000, so the execution condition is met.

[0124] Calculate the target and net quantity: The system uses the seventh calculation formula to calculate s(200) as approximately 19.0%, so the current target pruning total is 1,000,000 * 0.19 = 190,000.

[0125] At the previous update point t=100, s(100) was approximately 4.8%, so the total number of prunings for the previous objective was 48,000.

[0126] The current net number of prunings ΔN = 190,000 - 48,000 = 142,000.

[0127] Configuration adjustment quantity: The number of regrowth connections is configured to the preset number of handovers, which is 1,000.

[0128] The number of pruned connections is configured as the sum of ΔN and N_turnover, which is 142,000 + 1,000 = 143,000.

[0129] Through precise calculations in step B4, it was determined that at t=200, 143,000 connections need to be pruned and 1,000 connections need to be regenerated, thereby ensuring smooth growth of sparsity while continuously exploring the dynamic network structure.

[0130] Step B6: If the current training steps are greater than the total number of steps in the warm-up phase, then the target sparsity of the warm-up phase is used as the current weight connection sparsity and the current target pruning count corresponding to the current weight connection sparsity is determined. The preset handover count is used as the number of pruning connections and the number of regrowth connections for the target mask score tensor, respectively. When the current training steps *t* are greater than the total number of steps in the warm-up phase, the target sparsity of the warm-up phase is directly adopted as the current weight connection sparsity. This means that the target sparsity of the model will remain constant throughout the non-warm-up phase and will not increase further. Therefore, the corresponding total number of current target pruning steps will also remain unchanged, resulting in a net pruning count of 0 between the two updates.

[0131] Configure pruning and regrowth quantities: Based on the conclusion that the net pruning quantity is 0, the preset turnover quantity is configured as the number of pruning connections and the number of regrowth connections, respectively. That is: Number of pruning connections = Preset number of handovers Number of regenerated connections = preset number of trades This configuration ensures that in each mask adjustment, the number of connections cut off is exactly equal to the number of connections reactivated, thereby achieving dynamic updates and optimization of network connectivity while maintaining the overall sparsity.

[0132] Example: Using the content security audit service of an e-commerce platform as an example, and using the previous parameter settings: T_warmup=1000, S_target=0.8, N_turnover=1,000.

[0133] Scenario: During the 1500th training iteration (t=1500), the system triggered a mask adjustment.

[0134] Execution process: Condition judgment: t=1500 is greater than T_warmup=1000, so the execution condition is met.

[0135] Determine the target sparsity: Fix the current weighted connection sparsity to the target sparsity of the warm-up phase, which is 80%. Therefore, the total number of target prunings, both current and future, will remain constant at 1,000,000 * 0.8 = 800,000.

[0136] Configuration adjustment quantity: The system will configure the preset number of handovers (1,000) as the number of pruning and regrowth.

[0137] The number of regrowth connections is configured to 1,000.

[0138] The number of pruned connections is also configured to 1,000.

[0139] Through the calculation in step B6, the system determined that at t=1500, 1,000 currently active connections with the lowest importance need to be pruned, while simultaneously 1,000 currently inactive connections with the highest importance are regenerated. This operation achieves continuous "metabolism" and optimization of the network structure while maintaining 80% of the model's total sparsity.

[0140] S404: When the target mask score tensor is the dedicated residual mask score tensor, then based on the composite dynamic importance score of all model connection weight parameters, the scoring and ranking model connection weight step is performed on the dedicated residual mask score tensor. The scoring and ranking model connection weight step includes: sorting the model connection weight parameters of the active type to obtain an active model connection ranking set and sorting the model connection weight parameters of the inactive type to obtain an inactive model connection ranking set; selecting a first model connection weight parameter corresponding to the number of pruned connections from the active model connection ranking set in ascending order; setting the mask score tensor parameter corresponding to the first model connection weight parameter to a first type value of the inactive type; selecting a second model connection weight parameter corresponding to the number of pruned connections from the inactive model connection ranking set in descending order; and setting the mask score tensor parameter corresponding to the second model connection weight parameter to a second type value of the active type.

[0141] For a dedicated residual mask score tensor (designed for each computer processing task and independent of other tasks), a first reference value is set for the activation type, and model connection weight parameters with an aggregate importance score greater than or equal to the first reference value are determined as activation types; a second reference value is set for the inactivation type, and model connection weight parameters with an aggregate importance score less than or equal to the second reference value are determined as inactivation types. S406: When the target mask score tensor is the shared basic mask score tensor, the composite dynamic importance scores of all model connection weight parameters are weighted and aggregated to obtain an aggregated importance score. Based on the aggregated importance score of all model connection weight parameters, the scoring and sorting model connection weights step is performed on the shared basic mask score tensor. In one feasible implementation, for the base mask with a low update frequency, an index reflecting global importance is set. That is, the aggregated importance score is obtained by weighting and aggregating the composite dynamic importance scores of all model connection weight parameters, which can be done in the following way: The composite importance score is obtained by weighting and aggregating the composite dynamic importance scores of all model connection weight parameters using the eighth calculation formula. The eighth calculation formula satisfies the following formula:

[0142] Among them, the For the importance score of the aggregation, the This is a composite dynamic importance score for the i-th model connection weight parameter for the k-th computer processing task, where K is the total number of computer processing tasks.

[0143] The scoring and ranking model connection weight step includes: sorting the model connection weight parameters of the active type to obtain an active model connection ranking set and sorting the model connection weight parameters of the inactive type to obtain an inactive model connection ranking set; selecting a first model connection weight parameter corresponding to the number of pruned connections from the active model connection ranking set in ascending order; setting the mask score tensor parameter corresponding to the first model connection weight parameter to a first type value of the inactive type; selecting a second model connection weight parameter corresponding to the number of pruned connections from the inactive model connection ranking set in descending order; and setting the mask score tensor parameter corresponding to the second model connection weight parameter to a second type value of the active type.

[0144] For the shared base mask score tensor (shared by all computer processing tasks), a first threshold is set for activation types, and model connection weight parameters with aggregate importance scores greater than or equal to the first threshold are determined as activation types; a second threshold is set for inactivation types, and model connection weight parameters with aggregate importance scores less than or equal to the second threshold are determined as inactivation types. In this specification, a hierarchical mask dynamic update strategy is implemented in the manner described above. This periodic "pruning-regeneration" mechanism is used to dynamically adjust the mask. The basic mask, representing shared knowledge, is updated at a lower frequency to maintain stability, while the residual mask, representing task-specific knowledge, is updated at a higher frequency to quickly adapt to the respective task requirements.

[0145] Optionally, in one feasible implementation, the determination of the model's composite total loss based on the task loss of each of the computer processing tasks, the shared base mask score, the task-specific residual mask score, and the composite dynamic importance score can be performed in the following manner: The total task loss, global sparsity loss, and inter-task sparsity loss are determined based on the task loss of each computer processing task and the composite dynamic importance score. A composite total loss model is generated based on the total task loss, the global sparsity loss, and the inter-task sparsity loss. The total composite loss of the model satisfies the following formula in the ninth equation:

[0146] Among them, the The total composite loss of the model is... The total task loss is the weighted average of the task losses of all the tasks processed by the computers. For the global sparsity loss, the The sparse loss between the tasks is given, where and It is a hyperparameter that controls the strength of regularization. The total task loss satisfies the following formula in the tenth calculation:

[0147] Among them, the The task loss weight for the k-th computer processing task is... The task loss for the k-th computer processing task; in some embodiments, the task loss weight. Usable The global sparsity loss satisfies the following eleventh formula:

[0148] Among them, the To achieve the desired average activation density, For sparse schedulers The current weighted connection sparsity is generated. In some embodiments, the global sparsity loss can be defined as L2 loss, which indirectly adjusts all mask scores through gradients to guide the overall sparsity towards a preset trajectory. The expected average activation density satisfies:

[0149] N is the total number of connection weight parameters in the model, and K is the total number of tasks processed by the computer. Let the weight parameters of the i-th model connection weight parameters be the final activation probability for the k-th computer processing task, and the final activation probability satisfy:

[0150] The The shared fundamental activation probability of the i-th shared fundamental mask score tensor and satisfying The The residual activation probability of the residual mask score tensor for the i-th task and satisfying the following conditions: The For the i-th shared basic mask score tensor, the This is a task-specific residual mask score tensor; The inter-task sparsity loss satisfies the following formula in the twelfth equation:

[0151] Wherein, q0 is a preset constant. () denotes the Bernoulli distribution function, and DKL() denotes the KL divergence function: For example, q0 is a very small constant, such as 0.01; The inter-task sparsity loss regularization term is a key inter-task sparse structure consistency regularization term. It works by minimizing the activation probabilities of all residuals. The Bernoulli distribution and an extremely sparse prior Bernoulli distribution (in It is a very small constant, such as 0.01), between the KL divergence and the activation residual connection, which imposes a penalty. This regularization term applies to all... A strong gradient pressure is applied, pushing it toward negative infinity, thereby encouraging the model to preferentially reuse shared underlying connections.

[0152] Indicatively, in the backpropagation stage of each iteration, the entire learning process is unified under a single composite loss function. Within the optimization framework, refer to the ninth calculation formula. This loss function connects all the preceding steps, applying gradient descent to all learnable parameters (weights). And all mask scores Perform joint optimization. Calculate the total loss. Then, its gradient with respect to all learnable parameters is calculated using an automatic differentiation framework, and an optimizer (such as AdamW) is used for an update. This final optimization step not only updates the model weights. In order to improve task performance, and more importantly, it continuously fine-tunes the mask score through gradients, which complements the periodic large-scale adjustment in step three, together shaping a dynamically evolving, structurally optimized, high-performance and highly sparse hierarchical multi-task processing model.

[0153] To better understand the solution described in this specification, the following examples illustrate the application of the multi-task processing model involved in the training method of the above-mentioned multi-task processing model: Taking a specific scenario in the field of intelligent driving as an example, the scenario involves an in-vehicle computing platform that needs to deploy a single deep neural network model to simultaneously perform three key visual perception tasks: object detection, drivable region segmentation, and lane detection. A lightweight EfficientNet-B0 is chosen as the backbone network, followed by a multi-task architecture containing an ASPP module and three task-specific decoders. The dataset used is a large multi-task driving dataset such as BDD100K. We apply the H-TADS method of this invention to all convolutional and fully connected layers of the entire model, aiming to train a high-precision and highly sparse model to adapt to the limited computing resources of the in-vehicle platform.

[0154] In the implementation of the training method based on the multi-task processing model, a shared base mask score is initialized for the model. And the residual mask scores specific to the three tasks The training process strictly follows the training methodology for multi-task processing models: In each iteration, a dedicated computational graph is dynamically generated for each task, and forward propagation is performed to calculate the loss. Then, backpropagation is used to calculate the importance score of each weight for each task. Next, according to a preset period, these scores are used to perform pruning-regeneration operations on the base mask and residual masks to evolve the network topology. Finally, a unified gradient optimization is performed on the model weights and all mask scores under a composite loss function that includes dynamic task weighting (using the DWA algorithm), sparsity regularization, and structural consistency regularization. The final deployed model includes shared weights, a static base mask, and three residual masks. When a specific task needs to be performed, the system combines the corresponding masks in real time to build and run a dedicated sparse sub-network for that task.

[0155] This implementation yielded a multi-task perception model with deep structure optimization. Compared to traditional dense multi-task models, this model maintains comparable performance across perception tasks (such as object detection accuracy and segmentation crossover ratio) while achieving a highly sparse network structure. This directly leads to a significant reduction in computational complexity, enabling faster inference speeds on resource-constrained automotive hardware. Simultaneously, the model's storage requirements and runtime memory footprint are also substantially reduced. More importantly, through its hierarchical sparse structure, the model effectively isolates potential conflicts between tasks while promoting the sharing of common features, thereby enhancing the overall system's robustness and generalization ability, providing solid technical support for achieving safe and efficient autonomous driving.

[0156] This paper takes a specific application in medical image analysis as an example to illustrate the application of a multi-task processing model-based training method in the field of medical image analysis. The scenario is a medical AI-assisted diagnostic system that needs to process 3D images from different modalities (such as CT and MRI) and simultaneously perform lung nodule detection (based on CT) and brain tumor segmentation (based on MRI). We choose a powerful 3D Swin Transformer as the backbone network, followed by two 3D decoders specifically for detection and segmentation tasks. The training data comes from two independent public datasets, LUNA16 and BraTS, which are used together during training. The multi-task processing model-based training method is applied to all key linear transformation layers of the Swin Transformer, with the goal of generating a unified model that can handle multimodal data and efficiently perform multiple diagnostic tasks.

[0157] During implementation, a shared base mask score is initialized for the multi-task processing model. And the residual mask scores specific to the two tasks The training process follows the core four-step method of this invention: dynamically generating a sparse computation graph for a specific task for forward computation; evaluating the importance of each connection to each diagnostic task based on gradient and weight information; periodically evolving the base and residual mask structures, adjusting the shared knowledge base and task-specific knowledge modules; and finally, optimizing the model parameters and sparse structure end-to-end under the joint supervision of the dynamically weighted task loss and structure regularization loss using DWA. During deployment, depending on the input image type (CT or MRI), the corresponding residual mask is loaded and combined with the base mask to dynamically construct a dedicated sparse subnetwork for lung nodule detection or brain tumor segmentation.

[0158] This implementation successfully constructed an efficient and accurate multimodal medical diagnostic model. This multi-task processing model, through a hierarchical sparse structure, learns a powerful shared feature representation (defined by a base mask) applicable to different 3D medical images, while simultaneously developing highly specialized supplementary diagnostic logic (defined by a residual mask) for each specific diagnostic task. Compared to training a model separately for each task or using a simple dense multi-task model, the model obtained by this method exhibits superior performance on both diagnostic tasks, effectively avoiding the negative transfer problem caused by data heterogeneity. The model's high sparsity significantly accelerates the processing speed of massive 3D medical image data, shortens the diagnostic process, and reduces dependence on high-performance computing hardware. This capability enables high-quality AI-assisted diagnostic services to be more widely deployed in primary healthcare institutions, possessing significant clinical application value.

[0159] Compared with related technologies, the solutions of one or more embodiments in this specification have the following significant advantages: 1. Effectively mitigates negative transfer and improves model performance: Through a hierarchical sparse structure, common knowledge among tasks is shared in the basic sub-network, while conflicting and specific knowledge is learned in independent residual sub-networks. This fundamentally isolates gradient conflicts between different tasks, thereby improving the performance of each task.

[0160] 2. Achieve extremely high model compression and inference acceleration: This method can maintain or even surpass the performance of dense multi-task models while achieving extremely high sparsity. During inference, only the sparse subnetwork corresponding to the specific task needs to be loaded and computed, greatly reducing parameter storage and computation, making it particularly suitable for resource-constrained edge devices.

[0161] 3. Enhance model interpretability: After training, by analyzing the structure of the base mask and each residual mask, it is possible to intuitively understand which neurons are general feature extractors shared by all tasks and which are expert neurons serving specific tasks, providing a new perspective for model analysis.

[0162] 4. Excellent scalability: When a new task needs to be added to the system, the already trained shared weights and base mask can be frozen, and a new, small-scale residual mask can be learned only for the new task. This greatly reduces the cost of incremental learning and effectively avoids catastrophic forgetting of old tasks.

[0163] The following will combine Figure 5 This specification provides a detailed description of the training apparatus for the multi-task processing model provided in the embodiments. It should be noted that... Figure 5 The training apparatus for the multi-task processing model shown is used to execute this specification. Figures 1-4 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-4 The example shown.

[0164] Please see Figure 5 This diagram illustrates the structure of a training apparatus for a multi-task processing model according to an embodiment of this specification. The training apparatus 1 for the multi-task processing model can be implemented as all or part of a device through software, hardware, or a combination of both. According to some embodiments, the training apparatus 1 for the multi-task processing model includes a model creation module 11 and a model training module 12, specifically used for: The model creation module 11 is used to determine multiple computer processing tasks of different task types under a computer service scenario, create an initial multi-task processing model for the computer service scenario, initialize the model connection weight parameters of the initial multi-task processing model, initialize a shared basic mask score tensor shared by all the computer processing tasks, and initialize a dedicated residual mask score tensor for each computer processing task. Model training module 12 is used to generate effective weights for each computer processing task based on the shared base mask score tensor and the specific residual mask score tensor of each computer processing task during model training, perform forward propagation processing on each computer processing task based on the effective weights to obtain the prediction result of the computer processing task, and determine the task loss of each computer processing task based on the prediction result of the computer processing task. Model training module 12 is used to determine the composite dynamic importance score of each connection weight in the model connection weight parameters for each computer processing task based on the determined gradient information of each computer processing task and the gradient information and the model connection weight parameters. In one feasible implementation, the step of generating effective task weights for each computer processing task based on the shared base mask score tensor and the specific residual mask score tensor of each computer processing task, and performing forward propagation processing on each computer processing task based on the effective task weights to obtain the computer processing task prediction result includes: Based on the shared basic mask score tensor and the dedicated residual mask score tensor of each computer processing task, a connection weight graph is generated for each computer processing task to obtain the task-specific relaxed residual mask and the task relaxed basic mask. Based on the task-specific relaxation residual mask and the task-specific relaxation base mask, a mask combination is performed on each of the computer processing tasks to obtain a task-specific computation mask. The task-specific computation mask is then applied to the model connection weight parameters to obtain the effective weight of each of the computer processing tasks. Finally, the initial multi-task processing model is controlled to perform forward propagation processing on each of the computer processing tasks based on the effective weight of the tasks to obtain the prediction result of the computer processing tasks.

[0165] In one feasible implementation, the step of calculating a connection weight graph for each computer processing task based on the shared base mask score tensor and the task-specific residual mask score tensor to obtain a task-specific relaxed residual mask and a task-specific relaxed base mask includes: Sample noise tensors with the same mask tensor dimension from a preset random probability distribution; Based on the noise tensor and the preset temperature hyperparameter, the shared basic mask score tensor is masked using the first calculation formula to obtain the task relaxation basic mask; Based on the noise tensor and the preset temperature hyperparameter, the second mask solution is performed on the second formula of the dedicated residual mask score tensor of each computer processing task to obtain the task-specific relaxation residual mask. The first calculation formula satisfies the following formula:

[0166] Among them, the For the i-th element of the task relaxation base mask, the For the i-th element of the shared basic mask score tensor, the For the i-th element of the noise tensor with respect to the shared underlying mask score tensor, the This refers to the preset temperature hyperparameter; The second calculation formula satisfies the following formula:

[0167] Among them, the The i-th element of the task-specific relaxation residual mask for the k-th computer processing task, For the i-th element of the dedicated residual mask score tensor of the k-th computer processing task, the Let i be the i-th element of the noise tensor for the k-th computer processing task.

[0168] In one feasible implementation, the step of determining the composite dynamic importance score of each connection weight in the model connection weight parameters for each of the computer processing tasks based on the determined gradient information of each of the computer processing tasks, and based on the gradient information and the model connection weight parameters, includes: Based on the task loss of each of the aforementioned computer processing tasks, gradient information is determined for the connection weight parameters of the model; Based on the gradient information and the model connection weight parameters, determine the weight magnitude score, gradient significance score, and structural stability score for each connection weight in the model connection weight parameters for each computer processing task; Based on the weight magnitude score, gradient significance score, and structural stability score, a composite dynamic importance score for each connection weight in the model connection weight parameters is determined for each of the computer processing tasks.

[0169] In one feasible implementation, the weight magnitude score satisfies the third calculation formula, the gradient significance score satisfies the fourth calculation formula, the structural stability score satisfies the fifth calculation formula, and the composite dynamic importance score satisfies the sixth calculation formula. The third calculation formula satisfies the following formula:

[0170] The fourth calculation formula satisfies the following formula:

[0171] Wherein, Bi is the gradient significance score of the i-th model connection weight parameter, and L... k To represent the task loss of each of the aforementioned computer processing tasks; The fifth calculation formula satisfies the following formula:

[0172] Wherein, Ci is the structural stability score of the i-th model connection weight in the model connection weight parameters, and the... The output distribution influence parameter of the model for the connection weights of the i-th model;

[0173] The The model outputs the distribution influence parameter corresponding to the current training step t. It is the attenuation factor; The six calculation formulas satisfy the following formula:

[0174] Among them, the For the composite dynamic importance score, These are the preset hyperparameters and satisfy... .

[0175] In one feasible implementation, the model connection weight mask adjustment based on the composite dynamic importance score for the shared base mask score tensor and the specific residual mask score tensor for each of the computer processing tasks includes: Determine the target mask score tensor corresponding to the current training step, and determine the number of pruned connections and the number of regrowth connections for the target mask score tensor; When the target mask score tensor is the dedicated residual mask score tensor, then based on the composite dynamic importance score of all model connection weight parameters, the scoring and ranking model connection weight step is performed on the dedicated residual mask score tensor. When the target mask score tensor is the shared base mask score tensor, the composite dynamic importance scores of all model connection weight parameters are weighted and aggregated to obtain an aggregated importance score. Based on the aggregated importance score of all model connection weight parameters, a score ranking model connection weight step is performed on the shared base mask score tensor. The scoring and ranking model connection weight step includes: sorting the model connection weight parameters of the active type to obtain an active model connection ranking set and sorting the model connection weight parameters of the inactive type to obtain an inactive model connection ranking set; selecting a first model connection weight parameter corresponding to the number of pruned connections from the active model connection ranking set in ascending order; setting the mask score tensor parameter corresponding to the first model connection weight parameter to a first type value of the inactive type; selecting a second model connection weight parameter corresponding to the number of pruned connections from the inactive model connection ranking set in descending order; and setting the mask score tensor parameter corresponding to the second model connection weight parameter to a second type value of the active type.

[0176] In one feasible implementation, determining the target mask score tensor corresponding to the current training step, and determining the number of pruned connections and the number of regrowth connections for the target mask score tensor, includes: Based on the preset base mask training period and the current training step, determine whether the target mask score tensor corresponding to the current training step is a shared base mask score tensor; based on the preset dedicated mask training period and the current training step, determine whether the target mask score tensor corresponding to the current training step is a dedicated residual mask score tensor; If the current training step is less than or equal to the total number of steps in the warm-up phase, the seventh calculation formula is used to determine the current weight connection sparsity and the current target pruning total number corresponding to the current weight connection sparsity. Based on the target pruning total number and the previous target pruning total number corresponding to the previous training step, the current net pruning number is obtained. Based on the current net pruning number and the preset handover number, the number of pruned connections for the target mask score tensor is configured as the sum of the current net pruning number and the preset handover number, and the number of regrowth connections for the target mask score tensor is configured as the preset handover number. If the current training steps are greater than the total number of steps in the warm-up phase, then the target sparsity of the warm-up phase is used as the current weight connection sparsity and the current target pruning count corresponding to the current weight connection sparsity is determined. The preset number of hands is used as the number of pruning connections and the number of regrowth connections for the target mask score tensor, respectively. The seventh calculation formula satisfies the following formula:

[0177] Among them, the The target sparsity is for the preheating stage. The total number of steps in the warm-up phase, t is the current training step number, and the... Connect the sparsity of the current weights; The weighted aggregation of the composite dynamic importance scores of all model connection weight parameters yields an aggregated importance score, including: The composite importance score is obtained by weighting and aggregating the composite dynamic importance scores of all model connection weight parameters using the eighth calculation formula. The eighth calculation formula satisfies the following formula:

[0178] Among them, the For the importance score of the aggregation, the This is a composite dynamic importance score for the i-th model connection weight parameter for the k-th computer processing task, where K is the total number of computer processing tasks.

[0179] In one feasible implementation, the determination of the model's composite total loss based on the task loss of each of the computer processing tasks, the shared base mask score, the task-specific residual mask score, and the composite dynamic importance score includes: The total task loss, global sparsity loss, and inter-task sparsity loss are determined based on the task loss of each computer processing task and the composite dynamic importance score. A composite total loss model is generated based on the total task loss, the global sparsity loss, and the inter-task sparsity loss. The total composite loss of the model satisfies the following formula in the ninth equation:

[0180] Among them, the The total composite loss of the model is... The total task loss is the weighted average of the task losses of all the tasks processed by the computers. For the global sparsity loss, the This refers to the sparse loss between the tasks; The total task loss satisfies the following formula in the tenth calculation:

[0181] Among them, the The task loss weight for the k-th computer processing task is... The task loss for the kth computer-processed task; The global sparsity loss satisfies the following eleventh formula:

[0182] Among them, the To achieve the desired average activation density, the Given the current weighted connection sparsity, the expected average activation density satisfies:

[0183] N is the total number of connection weight parameters in the model, and K is the total number of tasks processed by the computer. Let the weight parameters of the i-th model connection weight parameters be the final activation probability for the k-th computer processing task, and the final activation probability satisfy:

[0184] The The shared fundamental activation probability of the i-th shared fundamental mask score tensor and satisfying The The residual activation probability of the residual mask score tensor for the i-th task and satisfying the following conditions: The For the i-th shared basic mask score tensor, the This is a task-specific residual mask score tensor; The inter-task sparsity loss satisfies the following formula in the twelfth equation:

[0185] Wherein, q0 is a preset constant. () denotes the Bernoulli distribution function, and DKL() denotes the KL divergence function.

[0186] It should be noted that the multi-task processing model device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the multi-task processing model method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the multi-task processing model device and the multi-task processing model method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0187] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0188] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-4The multi-task processing model method of the illustrated embodiment can be found in the following document for its specific execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0189] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-4 The multi-task processing model method of the illustrated embodiment can be found in the following document for its specific execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0190] Please refer to Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 1010, a memory 1020, an input device 1030, an output device 1040, and a bus 1050. The processor 1010, memory 1020, input device 1030, and output device 1040 may be connected to each other via the bus 1050.

[0191] Processor 1010 may include one or more processing cores. Processor 1010 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 1020, and by calling data stored in memory 1020. Optionally, processor 1010 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 1010 may integrate one or more of a central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 1010 and may be implemented separately through a communication chip.

[0192] The memory 1020 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1020 may include non-transitory computer-readable storage medium. The memory 1020 may be used to store instructions, programs, code, code sets, or instruction sets.

[0193] The input device 1030 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 1040 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In this embodiment, the input device 1030 can be a temperature sensor for acquiring the operating temperature of the electronic device. The output device 1040 can be a speaker for outputting audio signals.

[0194] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WIFI) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0195] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0196] exist Figure 5 In the electronic device, the processor 1010 can be used to call a program stored in the memory 1020 and execute it to implement the training method of the multi-task processing model as described in the various method embodiments of this specification.

[0197] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0198] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A training method for a multi-task processing model, characterized in that, The method includes: Identify multiple computer processing tasks of different task types in a computer service scenario, create an initial multi-task processing model for the computer service scenario, initialize a shared basic mask score tensor shared by all the computer processing tasks for the model connection weight parameters of the initial multi-task processing model, and initialize a dedicated residual mask score tensor for each computer processing task. During model training, effective weights are generated for each computer processing task based on the shared base mask score tensor and the specific residual mask score tensor of each computer processing task. Forward propagation is performed on each computer processing task based on the effective weights to obtain the prediction result of the computer processing task. The task loss of each computer processing task is determined based on the prediction result of the computer processing task. Based on the determined gradient information of each of the aforementioned computer processing tasks, and based on the gradient information and the model connection weight parameters, a composite dynamic importance score for each connection weight in the model connection weight parameters for each of the aforementioned computer processing tasks is determined; Based on the composite dynamic importance score, the model connection weight mask is adjusted for the shared basic mask score tensor and the dedicated residual mask score tensor of each computer processing task; The model's composite total loss is determined based on the task loss of each computer processing task, the shared basic mask score tensor, the task-specific residual mask score tensor, and the composite dynamic importance score. Based on the model's composite total loss, the model connection weight parameters, the shared basic mask score, and the task-specific residual mask score are adjusted until the model training ends and a trained multi-task processing model is obtained.

2. The method according to claim 1, characterized in that, The process of generating effective task weights for each computer processing task based on the shared base mask score tensor and the dedicated residual mask score tensor for each computer processing task, and performing forward propagation processing on each computer processing task based on the effective task weights to obtain the computer processing task prediction result includes: Based on the shared basic mask score tensor and the dedicated residual mask score tensor of each computer processing task, a connection weight graph is generated for each computer processing task to obtain the task-specific relaxed residual mask and the task relaxed basic mask. Based on the task-specific relaxation residual mask and the task-specific relaxation base mask, a mask combination is performed on each of the computer processing tasks to obtain a task-specific computation mask. The task-specific computation mask is then applied to the model connection weight parameters to obtain the effective weight of each of the computer processing tasks. Finally, the initial multi-task processing model is controlled to perform forward propagation processing on each of the computer processing tasks based on the effective weight of the tasks to obtain the prediction result of the computer processing tasks.

3. The method according to claim 2, characterized in that, The process of generating a connection weight graph for each computer processing task based on the shared basic mask score tensor and the task-specific residual mask score tensor, to obtain a task-specific relaxed residual mask and a task-specific relaxed basic mask, includes: Sample noise tensors with the same mask tensor dimension from a preset random probability distribution; Based on the noise tensor and the preset temperature hyperparameter, the shared basic mask score tensor is masked using the first calculation formula to obtain the task relaxation basic mask; Based on the noise tensor and the preset temperature hyperparameter, the second mask solution is performed on the second formula of the dedicated residual mask score tensor of each computer processing task to obtain the task-specific relaxation residual mask. The first calculation formula satisfies the following formula: Among them, the For the i-th element of the task relaxation base mask, the For the i-th element of the shared basic mask score tensor, the For the i-th element of the noise tensor with respect to the shared underlying mask score tensor, the This refers to the preset temperature hyperparameter; The second calculation formula satisfies the following formula: Among them, the The i-th element of the task-specific relaxation residual mask for the k-th computer processing task, For the i-th element of the dedicated residual mask score tensor of the k-th computer processing task, the Let i be the i-th element of the noise tensor for the k-th computer processing task.

4. The method according to claim 1, characterized in that, The determination of gradient information based on each of the aforementioned computer processing tasks, and the determination of a composite dynamic importance score for each connection weight in the model connection weight parameters for each of the aforementioned computer processing tasks based on the gradient information and the model connection weight parameters, including: Based on the task loss of each of the aforementioned computer processing tasks, gradient information is determined for the connection weight parameters of the model; Based on the gradient information and the model connection weight parameters, determine the weight magnitude score, gradient significance score, and structural stability score for each connection weight in the model connection weight parameters for each computer processing task; Based on the weight magnitude score, gradient significance score, and structural stability score, a composite dynamic importance score for each connection weight in the model connection weight parameters is determined for each of the computer processing tasks.

5. The method according to claim 4, characterized in that, The weight magnitude score satisfies the third calculation formula, the gradient significance score satisfies the fourth calculation formula, the structural stability score satisfies the fifth calculation formula, and the composite dynamic importance score satisfies the sixth calculation formula. The third calculation formula satisfies the following formula: Among them, the Here, Ai is the i-th model connection weight among the model connection weight parameters, and Ai is the weight score of the i-th model connection weight among the model connection weight parameters. This indicates how to scale the values ​​in each dimension to... Normalization operation within an interval; The fourth calculation formula satisfies the following formula: wherein the Bi is a gradient significance score of an i-th model connection weight of a model connection weight parameter, the L k is a task loss representing the respective computer processing task; The fifth calculation formula satisfies the following formula: Wherein, Ci is the structural stability score of the i-th model connection weight in the model connection weight parameters, and the... The output distribution influence parameter of the model for the connection weights of the i-th model; The The model outputs the distribution influence parameter corresponding to the current training step t. It is the attenuation factor; The six calculation formulas satisfy the following formula: Among them, the For the composite dynamic importance score, These are the preset hyperparameters and satisfy... .

6. The method according to claim 1, characterized in that, The step of adjusting the model connection weight mask based on the composite dynamic importance score for the shared base mask score tensor and the specific residual mask score tensor for each computer processing task includes: Determine the target mask score tensor corresponding to the current training step, and determine the number of pruned connections and the number of regrowth connections for the target mask score tensor; When the target mask score tensor is the dedicated residual mask score tensor, then based on the composite dynamic importance score of all model connection weight parameters, the scoring and ranking model connection weight step is performed on the dedicated residual mask score tensor. When the target mask score tensor is the shared base mask score tensor, the composite dynamic importance scores of all model connection weight parameters are weighted and aggregated to obtain an aggregated importance score. Based on the aggregated importance score of all model connection weight parameters, a score ranking model connection weight step is performed on the shared base mask score tensor. The scoring and ranking model connection weight step includes: sorting the model connection weight parameters of the active type to obtain an active model connection ranking set and sorting the model connection weight parameters of the inactive type to obtain an inactive model connection ranking set; selecting a first model connection weight parameter corresponding to the number of pruned connections from the active model connection ranking set in ascending order; setting the mask score tensor parameter corresponding to the first model connection weight parameter to a first type value of the inactive type; selecting a second model connection weight parameter corresponding to the number of pruned connections from the inactive model connection ranking set in descending order; and setting the mask score tensor parameter corresponding to the second model connection weight parameter to a second type value of the active type.

7. The method according to claim 6, characterized in that, The step of determining the target mask score tensor corresponding to the current training step, and determining the number of pruned connections and the number of regrowth connections for the target mask score tensor, includes: Based on the preset base mask training period and the current training step, determine whether the target mask score tensor corresponding to the current training step is a shared base mask score tensor; based on the preset dedicated mask training period and the current training step, determine whether the target mask score tensor corresponding to the current training step is a dedicated residual mask score tensor; If the current training step is less than or equal to the total number of steps in the warm-up phase, the seventh calculation formula is used to determine the current weight connection sparsity and the current target pruning total number corresponding to the current weight connection sparsity. Based on the target pruning total number and the previous target pruning total number corresponding to the previous training step, the current net pruning number is obtained. Based on the current net pruning number and the preset handover number, the number of pruned connections for the target mask score tensor is configured as the sum of the current net pruning number and the preset handover number, and the number of regrowth connections for the target mask score tensor is configured as the preset handover number. If the current training steps are greater than the total number of steps in the warm-up phase, then the target sparsity of the warm-up phase is used as the current weight connection sparsity and the current target pruning count corresponding to the current weight connection sparsity is determined. The preset number of hands is used as the number of pruning connections and the number of regrowth connections for the target mask score tensor, respectively. The seventh calculation formula satisfies the following formula: Among them, the The target sparsity is for the preheating stage. The total number of steps in the warm-up phase, t is the current training step number, and the... Connect the sparsity of the current weights; The weighted aggregation of the composite dynamic importance scores of all model connection weight parameters yields an aggregated importance score, including: The composite importance score is obtained by weighting and aggregating the composite dynamic importance scores of all model connection weight parameters using the eighth calculation formula. The eighth calculation formula satisfies the following formula: Among them, the For the importance score of the aggregation, the This is a composite dynamic importance score for the i-th model connection weight parameter of the k-th computer processing task, where K is the total number of computer processing tasks. The task loss weight is the weight for the k-th computer processing task.

8. The method according to claim 1, characterized in that, The method for determining the model's composite total loss based on the task loss of each of the aforementioned computer processing tasks, the shared basic mask score, the task-specific residual mask score, and the composite dynamic importance score includes: The total task loss, global sparsity loss, and inter-task sparsity loss are determined based on the task loss of each computer processing task and the composite dynamic importance score. A composite total loss model is generated based on the total task loss, the global sparsity loss, and the inter-task sparsity loss. The total composite loss of the model satisfies the following formula in the ninth equation: Among them, the The total composite loss of the model is... The total task loss is the weighted average of the task losses of all the tasks processed by the computers. For the global sparsity loss, the For the sparse loss between the tasks, the and These are hyperparameters; The total task loss satisfies the following formula in the tenth calculation: Among them, the The task loss weight for the k-th computer processing task is... The task loss for the kth computer-processed task; The global sparsity loss satisfies the following formula in the eleventh equation: Among them, the To achieve the desired average activation density, the Given the current weighted connection sparsity, the expected average activation density satisfies: N is the total number of connection weight parameters in the model, and K is the total number of tasks processed by the computer. Let the weight parameters of the i-th model connection weight parameters be the final activation probability for the k-th computer processing task, and the final activation probability satisfy: The The shared fundamental activation probability of the i-th shared fundamental mask score tensor and satisfying The The residual activation probability of the residual mask score tensor for the i-th task and satisfying the following conditions: The For the i-th shared basic mask score tensor, the This is a task-specific residual mask score tensor; The inter-task sparsity loss satisfies the following formula in the twelfth equation: Wherein, q0 is a preset constant. () denotes the Bernoulli distribution function, and DKL() denotes the KL divergence function.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as described in any one of claims 1 to 8.

Images