Training method and device of multitask model, electronic equipment and storage medium

Abstract

The application provides a multi-task model training method and device, an electronic device and a storage medium. The method comprises the following steps: obtaining training data of multiple tasks; generating an importance evaluation result of each attention head for each task according to a prediction result of a multi-task model based on the training data, and dividing the multiple tasks into multiple task sets according to the importance evaluation result; for each task set, selecting a target attention head based on the importance evaluation result, and constructing a corresponding mask matrix; performing gradient calculation based on the training data of the target task set to obtain a first gradient of a multi-layer perception module and a second gradient of each attention head; performing global parameter updating on the multi-layer perception module according to the first gradient, and performing local parameter updating on the attention module according to the mask matrix and the second gradient to obtain a target multi-task model. The application can effectively isolate the parameter interference between tasks and improve the comprehensive prediction accuracy of the multi-task model for different tasks.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method and apparatus for a multi-task model, an electronic device, and a storage medium. Background Technology

[0002] Multi-task learning (MTL) and instruction fine-tuning have become standard paradigms for adapting deep neural network models such as Large Language Models (LLMs). MTL fine-tunes a single model using mixed training data from multiple different prediction tasks, aiming to capture the potential connections between different tasks and thus improve the model's generalization ability and average performance. In practical applications, MTL combined with instruction fine-tuning frameworks can be used to build multi-domain AI assistants, enabling them to handle complex and diverse tasks such as commonsense reasoning, mathematical calculations, and code generation simultaneously.

[0003] Multi-task joint training methods in related technologies assume that different prediction tasks share parameters and intermediate representations, and update the model parameters uniformly. However, differences exist in parameter updates between different prediction tasks, and this approach cannot isolate parameter interference between different prediction tasks, leading to a significant deterioration in model performance. Therefore, how to improve the comprehensive prediction accuracy of multi-task models for different tasks has become an urgent problem to be solved. Summary of the Invention

[0004] The main objective of this application is to propose a training method and apparatus, electronic device and storage medium for a multi-task model, which aims to improve the comprehensive prediction accuracy of the multi-task model for different tasks.

[0005] To achieve the above objectives, a first aspect of this application proposes a training method for a multi-task model, wherein the multi-task model includes an attention module and a multilayer perceptron module, the attention module including multiple attention heads, and the method includes: Acquire training data for multiple tasks; For each task, an importance assessment result for each attention head to the task is generated based on the prediction result obtained by the multi-task model based on the training data. Based on all the importance assessment results, the multiple tasks are divided into multiple task sets; Based on the importance assessment results corresponding to each of the task sets, the corresponding target attention head is obtained by filtering from the multiple attention heads; A corresponding mask matrix is ​​constructed based on the target attention head corresponding to each task set. The mask value representation parameter of the target attention head in the mask matrix is ​​activated, and the mask value representation parameter of attention heads other than the target attention head is frozen in the mask matrix. The target training data of the target task set is extracted from the training data, and gradient calculation is performed based on the target training data by the multi-task model to obtain the first gradient of the multilayer perceptron module and the second gradient of each attention head. The multilayer perceptron module is globally updated based on the first gradient, and the attention module is locally updated based on the mask matrix corresponding to the target task set and the second gradient, to obtain the trained target multi-task model.

[0006] To achieve the above objectives, a second aspect of this application provides a training apparatus for a multi-task model, the multi-task model including an attention module and a multilayer perceptron module, the attention module including multiple attention heads, and the apparatus comprising: The acquisition unit is used to acquire training data for multiple tasks; The generation unit is configured to generate an importance evaluation result for each attention head to the task based on the prediction result obtained by the multi-task model based on the training data for each task. A partitioning unit is used to partition the multiple tasks based on all the importance evaluation results to obtain multiple task sets; A filtering unit is used to filter the multiple attention heads to obtain the corresponding target attention head based on the importance evaluation result corresponding to each task set; The construction unit is used to construct a corresponding mask matrix based on the target attention head corresponding to each task set. The mask value representation parameter of the target attention head in the mask matrix is ​​activated, and the mask value representation parameter of attention heads other than the target attention head is frozen in the mask matrix. The computing unit is used to extract target training data of the target task set from the training data, and to perform gradient calculation based on the target training data through the multi-task model to obtain the first gradient of the multilayer perceptron module and the second gradient of each attention head. The update unit is used to update the global parameters of the multilayer perceptron module according to the first gradient, and to update the local parameters of the attention module according to the mask matrix corresponding to the target task set and the second gradient, so as to obtain the trained target multi-task model.

[0007] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0008] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0009] This application proposes a training method, apparatus, electronic device, and storage medium for a multi-task model. First, it quantifies the importance assessment results of each attention head to each task and, based on this, divides tasks with similar model structural features into multiple task sets. Further, for each task set, it selects high-contribution target attention heads and constructs corresponding mask matrices to achieve structured parameter activation and freezing. During model iteration, a combined global and local update strategy is adopted. This retains the update method for shared module parameters to ensure the model's basic generalization ability, while using the mask matrix to restrict the parameter updates of attention modules to the target attention heads corresponding to specific task sets. Compared to the indiscriminate uniform parameter update method in related technologies, this application accurately isolates gradient conflicts between different prediction tasks, reduces parameter interference and negative transfer problems between heterogeneous tasks, thereby improving the overall prediction accuracy of the multi-task model when handling different tasks. Attached Figure Description

[0010] Figure 1 This is a flowchart of the training method for the multi-task model provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3 yes Figure 1 The flowchart of step S103 in the process; Figure 4 yes Figure 1 The flowchart of step S104 in the process; Figure 5 yes Figure 1 The flowchart of step S105 in the process; Figure 6 yes Figure 1 The flowchart of step S107 in the process; Figure 7 This is another flowchart of the training method for the multi-task model provided in the embodiments of this application; Figure 8 This is a schematic diagram of the flowchart of the training method for the multi-task model provided in the embodiments of this application; Figure 9 This is a comparison chart of the accuracy of this embodiment and related technologies on a large language model; Figure 10 This is a comparison chart of the generalization capabilities within the task cluster in this embodiment; Figure 11This is a schematic diagram of the structure of the training device for the multi-task model provided in the embodiments of this application; Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0012] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0014] Multi-task learning (MTL) and instruction fine-tuning have become standard paradigms for adapting deep neural network models such as Large Language Models (LLMs). MTL fine-tunes a single model using mixed training data from multiple different prediction tasks, aiming to capture the potential connections between different tasks and thus improve the model's generalization ability and average performance. In practical applications, MTL combined with instruction fine-tuning frameworks can be used to build multi-domain AI assistants, enabling them to handle complex and diverse tasks such as commonsense reasoning, mathematical calculations, and code generation simultaneously.

[0015] In practical applications, multi-task joint training methods in related technologies assume that different prediction tasks share parameters and intermediate representations, and uniformly update the model parameters. This approach ignores the task-specific structural features within the model (i.e., task-specific knowledge loops). Because parameter updates differ between different prediction tasks, simple joint training often leads to negative transfer problems when dealing with heterogeneous tasks. Gradients from different tasks may push shared parameters in incompatible directions, causing model performance to be even lower than that of single-task fine-tuning.

[0016] Even with efficient fine-tuning methods like LoRA, this problem persists. This is because strict low-rank constraints limit the optimization subspace, which may exacerbate gradient conflicts between unrelated tasks. Forcing incompatible tasks to share attention heads may further degrade model performance.

[0017] Therefore, the relevant technical methods cannot isolate the parameter interference between different prediction tasks, resulting in low overall prediction accuracy of multi-task models for different tasks.

[0018] Based on this, embodiments of this application provide a training method and apparatus for a multi-task model, an electronic device and a storage medium, aiming to improve the comprehensive prediction accuracy of the multi-task model for different tasks.

[0019] The training method, apparatus, electronic device, and storage medium for the multi-task model provided in this application are specifically described through the following embodiments. First, the training method for the multi-task model in this application is described.

[0020] The multi-task model training method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the multi-task model training method, but is not limited to the above forms.

[0021] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0022] Figure 1 This is an optional flowchart of the training method for the multi-task model provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0023] Step S101: Obtain training data for multiple tasks; Step S102: For each task, based on the prediction results obtained by the multi-task model from the training data, generate an evaluation result of the importance of each attention point to the task. Step S103: Based on all importance assessment results, the multiple tasks are divided to obtain multiple task sets; Step S104: Based on the importance evaluation results corresponding to each task set, select the corresponding target attention head from multiple attention heads; Step S105: Construct a corresponding mask matrix based on the target attention head for each task set. The mask value representation parameter of the target attention head in the mask matrix is ​​activated, and the mask value representation parameter of the attention heads other than the target attention head in the mask matrix is ​​frozen. Step S106: Extract target training data of the target task set from the training data, and perform gradient calculation based on the target training data using a multi-task model to obtain the first gradient of the multilayer perceptron module and the second gradient of each attention head. Step S107: Update the global parameters of the multilayer perceptron module according to the first gradient, and update the local parameters of the attention module according to the mask matrix corresponding to the target task set and the second gradient, to obtain the trained target multi-task model.

[0024] Before introducing the training method of the multi-task model in this application, let's first introduce the multi-task model itself. A multi-task model is a machine learning model that can handle multiple related tasks simultaneously. By sharing some parameters or structures and utilizing the commonalities between different tasks, a multi-task model can improve overall performance and generalization ability. Multi-task models can include large language models.

[0025] The underlying network architecture of the multi-task model includes an attention module and a multilayer perceptron (MLP) module. The attention module learns the relationships between different information in the input data, assigns different weights to different parts of the input data, and adjusts the model's focus on input features to concentrate on information more relevant to the current task. The attention module contains multiple independent attention heads, each responsible for capturing feature subspaces of different dimensions in the input data. Each attention head can independently learn different attention patterns, thereby enhancing the model's ability to handle complex relationships. The attention heads control the routing and inference patterns tightly coupled to a specific task or cluster of tasks. The multilayer perceptron module can include multiple fully connected layers. It performs nonlinear transformations and feature extraction on the features output by the attention module, generating the final prediction result. During multi-task fine-tuning, the multilayer perceptron module can perform global parameter updates to consolidate shared knowledge, while each independent attention head can selectively activate or lock parameters according to specific task requirements, thereby achieving precise optimization and parameter isolation for specific tasks.

[0026] Based on the aforementioned model architecture, the multi-task model has a wide range of application scenarios, such as building multi-domain all-around AI assistants based on large language models. In this application scenario, the AI ​​assistant simultaneously possesses the ability to handle multiple complex and heterogeneous tasks, such as common sense reasoning, mathematical calculation, and code generation. The multi-task model training method provided in this application can achieve seamless switching between different scenario functions without increasing the burden of additional reasoning computation. For example, the multi-task model can smoothly transition between answering historical common sense questions and writing code in a specific programming language (such as Python), and can effectively avoid the catastrophic forgetting phenomenon of forgetting existing common sense due to learning new domain knowledge such as code generation, ensuring that the multi-task model can maintain a high prediction accuracy when continuously learning different tasks.

[0027] In step S101, to achieve the comprehensive reasoning capabilities of a multi-task model (e.g., a large language model) across different domains, it is first necessary to construct training data for multi-task learning. Specifically, training data for multiple different tasks can be acquired. This training data includes input samples and their corresponding labels or target values ​​for the heterogeneous tasks that the multi-task model needs to master, such as commonsense reasoning tasks, mathematical calculation tasks, and code generation tasks. In practical applications, the training data for these multiple tasks can be collected and labeled manually. For example, in the field of natural language processing, text-label pairs for text classification tasks and text-entity pairs for named entity recognition tasks can be manually collected and labeled. This data is organized into a format suitable for model training. By learning from this data, the multi-task model can adjust its internal parameters to perform various tasks.

[0028] In step S102, to quantify the dependence of each component within the multi-task model on a specific task, ablation analysis of the fine-grained structure in the attention module is required. This embodiment characterizes the importance assessment result of the model structure features to the current task by calculating the performance degradation caused by the absence of a specific model structure. The importance assessment result measures the key role and dependence of each independent attention head within the model in obtaining the aforementioned prediction result. This embodiment, by comprehensively analyzing the performance of the prediction results, can objectively assess and quantify the supporting role of each attention head in completing a specific task. Specifically, the task loss value can be calculated without disabling any attention heads, then each attention head can be disabled one by one and the loss value recalculated. The importance of each attention head to the specific task can be assessed by comparing the changes in the loss value.

[0029] In some embodiments, the prediction result includes a first loss value and a second loss value. For each task, please refer to [reference needed]. Figure 2 Based on the prediction results obtained by the multi-task model from the training data, an importance evaluation result for each attention point to the task is generated, including: Step S201: Obtain the first loss value. The first loss value is obtained by the multi-task model through loss calculation based on the original model parameters and the training data corresponding to each task. Step S202: Perform a masking operation on each attention point to obtain the masked updated model parameters; Step S203: Obtain the second loss value. The second loss value is obtained by calculating the loss using the multi-task model based on mask-updated model parameters and the training data corresponding to each task. Step S204: For each task, generate an importance assessment result for each attention head to the task based on the first loss value and the second loss value.

[0030] In this embodiment, the technical problem of accurately and effectively quantifying the contribution of each attention head to a specific task needs to be addressed to ensure the accuracy of subsequent task partitioning and attention head selection. The prediction results obtained by the multi-task model based on training data include a first loss value and a second loss value. The first loss value represents the loss calculated by the multi-task model in its original state without any attention head masking, based on the original model parameters and the training data for the specific task. The first loss value characterizes the baseline performance of the multi-task model for the specific task in its original state. The loss calculation can employ methods such as cross-entropy loss, mean squared error loss, and KL divergence loss; the specific choice depends on the task type and is not limited here. The second loss value is obtained by calculating the loss through the multi-task model updating its model parameters based on the mask and the training data corresponding to each task. The second loss value reflects the performance of the multi-task model on the task after a specific attention head is masked. Similar to the first loss value, the calculation of the second loss value also uses a loss function that matches the task type.

[0031] In step S201, the baseline prediction performance of the model without any structured intervention is first obtained. Specifically, the multi-task model includes the original model parameters. , targeting Training datasets for different tasks, for each prediction task The training data corresponding to the task is input into the multi-task model for forward propagation calculation to obtain the prediction result. Subsequently, based on the difference between the prediction result and the true label, the first loss value is calculated. .

[0032] In step S202, to individually evaluate the role of a specific fine-grained structure in the multi-task model, this embodiment performs targeted masking processing on the model structure. The attention module in the multi-task model contains multiple layers (denoted as...). (layers) and each layer contains multiple attention heads (denoted as...) (Number). For a specific task, this embodiment traverses each attention head in the multi-task model, and for the first attention head... Layer Each attention head undergoes a masking operation. This masking operation can be achieved by setting the output weights or activation values ​​of a specific attention head to zero, thus rendering it ineffective during the model's forward propagation; alternatively, it can be achieved by replacing the attention head's output with a preset fixed value (e.g., average value or random noise). In this way, scenarios where attention heads are missing or ineffective can be simulated, allowing for masking and updating of model parameters. .

[0033] In step S203, after constructing the mask to update the model parameters, it is necessary to further evaluate the impact of missing local structures on the overall prediction performance of the model, and quantify the performance fluctuation caused by the masking of specific attention points. Specifically, the same task can be used... The training data is input into the model parameters updated using a mask. In the multi-task model, the model parameters are updated based on the mask, and the forward computation is performed again. The error difference between the predicted result and the true label is evaluated to obtain the second loss value. The second loss value reflects the specific attention head (i.e., the first...) Layer When an attention head is masked, the model exhibits the error level in handling the current prediction task. If the masking operation leads to a significant increase in the loss value (i.e., the second loss value is much larger than the first loss value), it indicates that the masked attention head is of high importance to the task.

[0034] In step S204, the specific contribution of each attention point to a specific task is calculated based on the first loss value and the second loss value, i.e., the importance assessment result. Specifically, for each task... The importance assessment result of the attention head to the current task is calculated by calculating the increase in loss caused by masking a specific attention head.

[0035] Furthermore, flatten and stitch together all attention heads in the multi-task model ( (Attention focus) on a specific task The importance assessment results are calculated and generated. dimensional embedding vector (where This allows for a comprehensive and accurate quantification of the dependence of a specific task on the various components (attention heads) within the model.

[0036] In steps S201 to S204 of this application, for each task and each attention head, the baseline performance, i.e., the first loss value, is first calculated in the complete state of the multi-task model. Then, by masking a single attention head, the model state when the attention head is missing is simulated, and a second loss value is calculated in this state. By comparing the first and second loss values, the independent contribution of each attention head to the performance of a specific task is quantitatively revealed. Thus, this embodiment can effectively avoid evaluation bias caused by a single prediction result and more accurately capture the correlation strength between the attention head and the task.

[0037] In step S103, to address the gradient conflicts and negative transfer issues arising during heterogeneous task hybrid training, this embodiment groups compatible tasks into the same task cluster. This ensures that tasks within the same cluster share overlapping inference paths within the multi-task model, avoiding parameter interference between incompatible tasks. Here, the task cluster refers to the set obtained by clustering multiple tasks based on similarity or importance evaluation results. Specifically, tasks can be grouped based on semantic similarity or the similarity of their activated attention head patterns in the model, grouping tasks exhibiting similar importance patterns on specific attention heads into one category.

[0038] In some embodiments, the importance assessment results include an importance score, please refer to... Figure 3 Based on the results of all importance assessments, multiple tasks are divided into several task sets, including: Step S301: Calculate the importance score of the attention head to the task based on the first loss value and the second loss value; Step S302: For each task, generate a task feature embedding vector based on all importance scores of the task. Step S303: Calculate the distance between the task feature embedding vectors corresponding to each pair of tasks, and divide the multiple tasks according to the distance values ​​to obtain multiple task sets.

[0039] In this embodiment, the importance assessment result includes an importance score, which is a quantitative indicator of the degree to which the attention head contributes to a specific task. The importance score can be determined by the difference between a first loss value and a second loss value. For example, when an attention head is masked, the greater the increase in the second loss value relative to the first loss value, the higher the importance of the attention head to the task. Alternatively, the importance score can also be determined by the ratio of the first loss value to the second loss value, or by mapping the two through a nonlinear function.

[0040] In step S301, the first loss value represents the baseline prediction error of the model under the original parameter state, and the second loss value represents the prediction error of the model after a specific attention head is masked. For each prediction task in the multi-task model, the difference between the first loss value and the second loss value is calculated to obtain the importance score of the corresponding attention head for that task. The specific formula for calculating the importance score is as follows: (1) in, The multi-task model is based on the original model parameters. The first loss value generated from the training data corresponding to the task. The multi-task model is based on masking the first... Layer After each attention point, the mask updates the model parameters. The second loss value is obtained. Through difference calculation, if the error increases significantly after a certain attention head is masked (i.e., the second loss value is much greater than the first loss value), then its corresponding importance score is high, indicating that the attention head has a high dependence and importance in completing the specific prediction task.

[0041] In step S302, after obtaining the importance scores of each attention head within the multi-task model for a specific task, these scores need to be integrated to compare the dependency preferences of the underlying model parameters for different tasks at a macro level. Specifically, for each prediction task, all attention heads in the multi-task model can be integrated. Each layer attention points (total) The importance score corresponding to each (item). The flattening and stitching operations are performed according to a fixed network structure order. Through the above stitching operation, a network with dimension [missing information] can be generated. The feature vector, i.e., the task feature embedding vector, where Task feature embedding vectors are numerical vectors used to characterize the behavioral patterns of a task in the attention module, depicting the dependencies of a single task on specific functional structures within a multi-task model.

[0042] In step S303, similarity calculations can be used to automatically categorize compatible tasks with overlapping underlying inference paths, providing a basis for task partitioning to address gradient conflicts and negative transfer issues in multi-task learning. To measure the compatibility between different prediction tasks, it is necessary to quantitatively calculate the distance between the task feature embedding vectors corresponding to each pair of tasks.

[0043] Specifically, in actual model training scenarios, all models containing... The feature embedding vectors of different prediction tasks are stacked to obtain an embedding matrix representing the overall task features. Subsequently, based on the embedding matrix This method employs specific distance metrics (such as cosine similarity or Euclidean distance) to calculate the distance between feature embedding vectors for different tasks, thereby measuring the functional consistency of different tasks in terms of attention head usage preferences. For example, cosine similarity can effectively evaluate the directional angle between two high-dimensional vectors in multidimensional space. If the distance value indicates that the two vectors are highly similar, it means that the two prediction tasks are highly dependent on the same attention head components within the model.

[0044] Furthermore, after calculating the distance between the feature embedding vectors of each task, an automated clustering algorithm (such as K-Medoids clustering) is used to divide the multiple prediction tasks. The clustering algorithm groups tasks that are close together (i.e., have high functional consistency and strong compatibility) into the same cluster, ultimately reorganizing the original clusters. The prediction tasks are divided into: A set of functionally compatible tasks, a task set (task cluster) can be represented as: , Represents a set of multiple tasks A set of tasks in the [context]. A dataset can contain multiple tasks. This ensures that prediction tasks within the same task set can effectively share specific model parameters during subsequent joint fine-tuning, achieving parameter isolation between different task sets.

[0045] Steps S301 to S303 of this embodiment accurately calculate the importance score of each attention head for each task by comparing the performance changes of the model under the original parameters and after a specific attention head is masked, quantifying the contribution of each attention head to completing a specific task. Next, for each task, its importance score is aggregated with the importance scores of all attention heads to generate a task feature embedding vector. This vector depicts the overall dependency pattern of the task on the attention module in a multi-dimensional space. Subsequently, the similarity of their usage patterns in the attention mechanism is quantified by calculating the distance between the task feature embedding vectors of any two tasks. Finally, multiple tasks are divided based on the calculated distance values, grouping tasks with similar attention patterns into the same task set. Thus, this embodiment ensures a high degree of consistency in the usage preferences of the attention module among tasks within each task set, ensuring high homogeneity of tasks within the set, providing more accurate guidance for subsequent local parameter updates, avoiding parameter interference between unrelated tasks, and improving the training effect of the multi-task model.

[0046] In step S104, a target attention head can be selected from multiple attention heads based on the importance evaluation results corresponding to each task set. A target attention head refers to one or more attention heads that are identified as making a significant contribution to the performance of a specific task set.

[0047] In some embodiments, please refer to Figure 4 Based on the importance evaluation results corresponding to each task set, the corresponding target attention head is selected from multiple attention heads, including: Step S401: For each task set, calculate the corresponding cluster importance score based on the number of tasks in the task set and the importance evaluation results of all tasks in the task set. Step S402: Based on the cluster importance score from high to low, select the corresponding preset number of target attention heads from multiple attention heads.

[0048] In step S401, for each task set, the corresponding cluster importance score is calculated based on the number of tasks in the task set and the importance assessment results of all tasks in the task set. The calculation of the cluster importance score takes into account the size of the task set (number of tasks) and the individual importance assessment results of all tasks in the set for each attention head. The cluster importance score can link the overall characteristics of the task set with the contribution of the attention head, avoiding reliance on the assessment of a single task, and thus more comprehensively measuring the importance of the attention head to the entire task set.

[0049] Specifically, for a given set of tasks, the first step is to determine the number of tasks contained in that set. The importance assessment results for all tasks in the task set are also considered. To eliminate the impact of differences in loss scale or data distribution between different tasks, the importance assessment results for individual tasks can be normalized. Subsequently, based on the number of tasks in the task set and the normalized importance assessment results for all tasks, the cluster importance score for each attention point corresponding to that task set is calculated. The specific formula for calculating the cluster importance score is as follows: (2) in, Represents a set of tasks Each independent task in the game, This is the importance assessment result after normalization. The above formula calculates the importance of all tasks within the task set for a specific attention head (the first attention head). Layer The average importance score of each attention head is used to quantify the cluster importance score of the attention head to the current task set as a whole.

[0050] In step S402, redundant attention heads that contribute little or no to the current task set can be further removed, isolating gradient conflicts between different heterogeneous task sets at the source. Specifically, multiple attention heads within the multi-task model can be sorted in descending order of cluster importance score. After sorting, a predetermined number of attention heads are selected as target attention heads from the above attention heads in descending order of cluster importance score. For example, for each attention layer of the multi-task model... You can select the cluster with the highest importance score. indivual( (This is the preset number) of attention heads. Finally, the target attention heads selected from all attention layers of the multi-task model are merged, and the selected target attention heads form cluster circuits for the corresponding task set.

[0051] In steps S401 and S402 of this embodiment, for each task set, the importance assessment result of the attention head is not solely based on the individual task's importance assessment result, but rather comprehensively considers the number of tasks included in the task set and the importance assessment results of all tasks within the set for each attention head. A clustering importance score is calculated to reflect the contribution of the attention head to the entire task set. This score can more comprehensively measure the overall value of the attention head when processing the task set. Subsequently, based on the clustering importance score, a predetermined number of target attention heads are selected from all available attention heads in descending order of importance. Thus, this embodiment overcomes the limitations that may arise from evaluating only a single task, making the resource allocation of the attention module more precise and efficient, thereby achieving better performance in multi-task learning.

[0052] In step S105, a corresponding mask matrix is ​​constructed based on the target attention head corresponding to each task set. The mask value in the mask matrix represents parameter activation or parameter freezing. The position corresponding to the target attention head is set to a value that allows parameter updates (e.g., 1), while the positions corresponding to other attention heads are set to values ​​that prohibit parameter updates (e.g., 0).

[0053] In some embodiments, please refer to Figure 5 Construct a mask matrix based on the target attention head for each task set, including: Step S501: Obtain the size parameters of the attention module; Step S502: Construct the corresponding mask matrix based on the target attention head and size parameters corresponding to each task set.

[0054] In step S501, during the training of the multi-task model, after determining the target attention head corresponding to each task set, a corresponding mask matrix needs to be constructed to achieve local parameter updates. However, if the structural characteristics of the attention module itself are not fully considered when constructing the mask matrix, the constructed mask matrix may be inaccurate, affecting the accuracy of parameter activation and freezing, thereby leading to low model training efficiency and decreased model performance. Multi-task models typically employ a multi-layer stacked network architecture, where the attention module contains a specific spatial dimension structure. The size parameters mainly include the number of layers in the attention mechanism of the multi-task model. and the number of independent attention heads contained in each layer. By obtaining the aforementioned size parameters, this embodiment can completely map the total within the multi-task model. The spatial topology of an attention head.

[0055] In step S502, after obtaining the size parameters and the target attention heads selected for the specific task set, a corresponding mask matrix can be constructed. The mask matrix is ​​used to limit the update range of model parameters during backpropagation, effectively isolating gradient conflicts between incompatible tasks.

[0056] Specifically, for each predefined task set (i.e., clustering) ), can be based on the number of floors and head number Two size parameters are used to initialize a global matrix that perfectly matches the spatial dimensions. Then, the selected task set is grouped into clustering loops (i.e., the set of target attention heads). The coordinates of the given coordinates are mapped to the aforementioned global matrix. Specifically, a binary mask matrix is ​​defined. Iterate through each spatial position defined by the size parameters, if the first... Layer Each attention head belongs to the target attention head of the task set (i.e., satisfies the coordinate condition). Then the binary mask matrix The mask value at the corresponding position in the middle The value is assigned to 1.

[0057] If the current iteration is the th Layer A target attention head that does not belong to a specific task set (i.e., does not belong to a clustering loop). Then the binary mask matrix The mask value at the corresponding position in the middle The value is assigned to 0. This embodiment constructs a binary mask matrix through precise assignment operations based on size parameters and the spatial distribution of the target attention head. The positions with a mask value of 1 represent parameter activation, allowing gradient backpropagation and normal updates; while the positions with a mask value of 0 represent parameter freezing, which can block updates from non-loop attention heads.

[0058] In steps S501 to S502 of this embodiment, the size parameters of the attention module are obtained and combined with the target attention heads selected for a specific task set to construct a corresponding mask matrix. This ensures that the generated mask matrix is ​​aligned with the actual parameter structure of the attention module, avoiding mask errors caused by structural mismatch. Thus, the masking mechanism provided in this embodiment enables multi-task models to more effectively perform local parameter updates during training, adjusting parameters only for attention heads highly relevant to the current task set while freezing the parameters of irrelevant attention heads. This helps prevent negative transfer between heterogeneous tasks, allowing the model to better learn and distinguish the features of different tasks, thereby improving the overall performance and generalization ability of the multi-task model.

[0059] In step S106, the multilayer perceptron module tends to encode globally transferable feature transformations, while the attention module controls the routing inference mode tightly coupled to a specific task cluster. In each training iteration, a mini-batch of data is randomly sampled from a randomly selected set of target tasks as the target training data. The multi-task model performs forward computation based on the target training data to derive the prediction error, and then calculates the current original gradients of all parameters through backpropagation. The gradient calculated for the parameters of the multilayer perceptron module is the first gradient. The first gradient is used to guide the parameter updates of the multilayer perceptron module, and the original gradient calculated for the parameters of all attention heads is the second gradient. The second gradient is used to guide the parameter updates of the attention module.

[0060] In step S107, a clustering gating optimization strategy is executed, allowing the general representation module to absorb knowledge from all tasks while restricting the parameter updates of the routing module to a compatible task subspace. This reduces the number of active parameters, effectively isolates gradient signals, and avoids parameter interference between different tasks. Specifically, the multilayer perceptron module can be globally updated based on the first gradient, and the attention module can be locally updated based on the mask matrix corresponding to the target task set and the second gradient, resulting in the trained target multi-task model.

[0061] In some embodiments, please refer to Figure 6 The attention module's local parameters are updated based on the mask matrix and second gradient corresponding to the target task set, resulting in the trained target multi-task model, including: Step S601: Multiply the mask matrix corresponding to the target task set element by element with each second gradient to obtain the target mask gradient; Step S602: Update the local parameters of the attention module based on the target mask gradient to obtain the trained target multi-task model.

[0062] In step S601, during the training of a multi-task deep learning model, the direction and magnitude of the gradient determine the trajectory of parameter updates. If the original second gradient is used to update the parameters of all attention heads without intervention, gradient conflicts between different heterogeneous tasks will occur, pushing shared parameters in incompatible directions and thus triggering a negative transfer problem.

[0063] To achieve gradient filtering, this embodiment performs element-wise multiplication of the mask matrix corresponding to a specific set of target tasks with the original second gradient calculated by each attention head in the attention module of the multi-task model. Specifically, let's assume the mask matrix corresponding to the set of target tasks is denoted as... The original second gradient of the attention module is denoted as Then, the element-wise multiplication operation can be represented as: The gradient of the target mask is calculated.

[0064] Based on the aforementioned rules for constructing mask matrices, since the mask matrix... The mask value for positions belonging to the target attention head is 1, while the mask value for positions not belonging to the target attention head is 0. Therefore, after element-wise multiplication, the second gradient originally targeting non-target attention head regions is multiplied by 0, and its gradient value is forced to 0; while the second gradient targeting the target attention head regions is multiplied by 1, and its original gradient value is completely preserved. The resulting target mask gradient accurately eliminates redundant gradient signals that cause mutual interference between incompatible tasks in the network structure, realizing dynamic gradient routing.

[0065] In step S602, this embodiment performs clustering gating optimization, breaking the inherent way of uniformly updating all model parameters in traditional multi-task joint training in related technologies. By implementing local parameter updates on the attention module, the routing updates of specific task sets are isolated, ensuring that the model will not forget existing knowledge due to severe parameter overwriting when continuously learning new tasks.

[0066] Specifically, the calculated target mask gradient is used in conjunction with the network optimizer algorithm to update the local parameters of the attention module in the multi-task model. The update iterative process based on the target mask gradient can be described as follows: ,in, This indicates the current parameters of the attention module. This refers to the pre-set hyperparameter learning rate used to control the step size of each parameter update. This represents the target mask gradient calculated in the preceding steps.

[0067] Local parameter updates refer to the iterative changes in parameters of only the regions activated by the mask matrix (i.e., regions with non-zero target mask gradients, corresponding to clustering loops of the task set) within the entire attention module's parameter set, occurring during backpropagation. Parameters in non-clustering loop regions are frozen in this iteration, maintaining their original weight values, as their corresponding target mask gradients have been forcibly set to zero. Through iteration with massive amounts of data and local parameter updates, this embodiment reduces the number of active parameters, effectively avoiding parameter interference problems in multi-task model fine-tuning, resulting in the trained target multi-task model. The trained target multi-task model can efficiently utilize its internally fixed independent attention head loops for accurate prediction when handling heterogeneous tasks with different clusters, such as commonsense reasoning and code generation.

[0068] In this embodiment, steps S601 to S602, by element-wise multiplying the mask matrix with the second gradient, ensure that only the gradients of the target attention heads related to the current target task set are preserved, while the gradients of other non-target attention heads are effectively set to zero. This masked target gradient can directly guide the optimizer to adjust only the parameters of the target attention heads in subsequent local parameter updates, thereby achieving refined and localized updates to the attention module. This not only avoids unnecessary interference to irrelevant attention head parameters, improving the targeting and efficiency of parameter updates, but also helps the model better learn and distinguish the features of different tasks, enhancing the overall performance and generalization ability of the multi-task model.

[0069] Steps S101 to S107, as illustrated in the embodiments of this application, firstly quantify the importance evaluation results of each attention head to each task, and accordingly divide tasks with similar model structure features into multiple task sets. Further, for each task set, high-contribution target attention heads are selected, and corresponding mask matrices are constructed to achieve structured parameter activation and freezing. During model iteration, a combined global and local update strategy is adopted. This retains the update method for shared module parameters to ensure the basic generalization ability of the model, while using the mask matrix to restrict the parameter updates of attention modules to the target attention heads corresponding to specific task sets. Compared to the indiscriminate uniform parameter update method in related technologies, this application accurately isolates gradient conflicts between different prediction tasks, reduces parameter interference and negative transfer problems between heterogeneous tasks, thereby improving the overall prediction accuracy of multi-task models when handling different tasks.

[0070] In some embodiments, please refer to Figure 7After updating the global parameters of the multilayer perceptron module based on the first gradient and updating the local parameters of the attention module based on the mask matrix corresponding to the target task set and the second gradient, to obtain the trained target multi-task model, the method of this application further includes: Step S701: Obtain the task to be predicted, determine the set of target tasks that match the task to be predicted, and determine the second target mask matrix corresponding to the set of target tasks; Step S702: Call the target multi-task model to generate the task prediction result of the task to be predicted based on the second target mask matrix.

[0071] In step S701, in practical application scenarios, in order to achieve seamless switching between multi-domain tasks without increasing the additional inference burden, and to avoid irrelevant knowledge interference caused by the model calling all attention mechanisms during inference, this embodiment can accurately match the new task input at the moment with the pre-trained and solidified local parameter loop (i.e., clustering loop).

[0072] Specifically, the task to be predicted can be obtained first. The task to be predicted refers to the specific problem or input data that the target multi-task model needs to handle in practical applications, such as common sense reasoning problems or code generation instructions received when building an all-around AI assistant. Then, the set of target tasks that match the task to be predicted can be determined. Specifically, it can be classified into a preset task set that is most functionally compatible and has the most consistent underlying reasoning path based on the task type or domain characteristics of the task to be predicted. For example, it can be classified by the semantic similarity between the task to be predicted and the tasks in the task set, or it can be divided into matching task sets by a pre-trained classification network. Alternatively, the task feature embedding vector of the task to be predicted can be calculated on a small number of samples, and its distance to the center of each existing task set can be calculated. It can then be divided into the target task set that is closest in distance (i.e., has the most consistent underlying reasoning path). There are no restrictions here.

[0073] After identifying the target task set to which the task to be predicted belongs, the second target mask matrix corresponding to the target task set is further determined. The second target mask matrix is ​​compared with the mask matrix (i.e., the binary mask matrix) constructed in the aforementioned training phase. Correspondingly, its size parameters match the attention module structure of the multi-task model. The second target mask matrix includes the position information of the target attention heads specific to the target task set. Positions with a mask value of 1 represent attention heads that need to be activated during inference, while positions with a mask value of 0 represent attention heads that need to be masked or ignored during inference.

[0074] In step S702, after determining the second target mask matrix that matches the task to be predicted, the target multi-task model can be invoked to generate the task prediction result based on the second target mask matrix. In the actual forward propagation computation, the task to be predicted first passes through the globally shared multilayer perceptron module to invoke the globally shared knowledge consolidated during the model training phase. When the data flows to the attention module, the multi-task model only activates the target attention heads corresponding to a mask value of 1 in the second target mask matrix to calculate attention weights and extract features, while attention heads with a mask value of 0 do not participate in the current inference computation (or their output is masked).

[0075] In this way, the target multi-task model can focus on the knowledge loops required to process the task to be predicted. This not only ensures that the reasoning logic learned from the training data can be effectively mapped to the new prediction task, promoting better positive transfer, but also further enhances the model's zero-shot generalization ability for unseen tasks within the same cluster (i.e., within the same set of target tasks). Finally, based on the aforementioned constrained reasoning path, the multi-task model outputs high-accuracy task prediction results, achieving efficient processing of complex and heterogeneous tasks.

[0076] In this embodiment, steps S701 to S702, after the multi-task model training is completed, dynamically select and activate the most relevant attention heads for a specific task to be predicted. This allows the model to more accurately focus on task-related features during the inference phase, avoiding interference from irrelevant attention heads. This embodiment not only improves the accuracy and efficiency of the model in handling specific tasks, but also, through a customized attention mechanism, fully utilizes the task-specific knowledge learned during training in practical applications, enhancing the model's generalization ability and adaptability, thereby generating high-quality prediction results for the task to be predicted. In some embodiments, please refer to Figure 8 , Figure 8 This is a schematic diagram of the flowchart framework of the training method for the multi-task model provided in the embodiments of this application, as shown below. Figure 8 As shown, for multiple different input tasks (task 1, task 2, task 3, etc.), the dependency features of each task within the model are extracted to generate corresponding task feature embedding vectors. To extract the task feature embedding vectors, this embodiment uses a specific attention head masking method specific to the multi-task model. Figure 8 The gray squares in the diagram represent masking specific attention heads. The difference in loss function before and after masking is calculated to quantify the importance of that attention head to a specific task, resulting in an importance score. Then, task feature embedding vectors are generated based on all the task's importance scores. Automatic task clustering can then be performed. Specifically, based on the extracted task embedding vectors, tasks with similar and compatible underlying inference paths are automatically grouped in the vector space, such as... Figure 8 As shown, tasks are divided into different clusters; for example, tasks 1 and 2 are assigned to task set 1, and tasks 3 and 4 are assigned to task set 2. Further, for each task set, the importance score is first normalized, and then target attention heads are selected, thereby constructing a corresponding clustering loop for each task set (clustering loop 1 for task set 1 and clustering loop 2 for task set 2), where the clustering loop represents the spatial mask matrix formed by the selected target attention heads.

[0077] Furthermore, clustering gating optimization is performed. When the multi-task model receives training data from tasks in task set 1, only the attention heads in clustering loop 1 are activated, while the attention heads in clustering loop 2 are locked. Simultaneously, the multilayer perceptron module receives data from all clusters, enabling global knowledge sharing and parameter updates. Finally, through a gating optimization strategy combining local attention isolation and global MLP sharing, the updated parameters are output, effectively avoiding negative transfer between incompatible tasks.

[0078] Please refer to Figure 9 , Figure 9 This is a comparison chart of the accuracy of this embodiment and related technologies on a large language model, such as... Figure 9 As shown, in eight commonsense reasoning benchmark tests conducted on the LLaMA3-8B and Qwen3-8B models (including WinoG (WinoGrande semantic disambiguation dataset), SIQA (Social Interaction Question Answering dataset), HellaS (HellaSwag commonsense reasoning dataset), BoolQ (Boolean question answering dataset), ARC-C (Allen Reasoning-Challenge-Challenge set), OBQA (Open Book Question Answering dataset), PIQA (Physical Interaction Question Answering dataset), and ARC-E (Allen Reasoning Challenge-Simple set)), this embodiment achieved higher average accuracy compared to baseline methods such as LoraRetriever, LoRI, single-task, joint learning, and random loops. When the number of clusters M=3, the average accuracy of LLaMA3-8B reached 87.09%, and that of Qwen3-8B reached 90.07%, significantly outperforming the standard joint learning baseline.

[0079] Please refer to Figure 10 , Figure 10 This embodiment demonstrates its performance in intra-cluster generalization. To test the generalization ability of the multi-task model, this embodiment employs a leave-one-out testing strategy. Specifically, after extracting the corresponding clustering circuits for each task cluster (i.e., the task set), one or more tasks are removed from the training data. The multi-task model is trained using only the remaining tasks within the task cluster, and finally, the trained model is tested on the unseen removed tasks (retained tasks).

[0080] Figure 10 The blue area (Task Set 1: Science Common Sense Tasks) indicates that the model was trained only on a subset of tasks including ARC-E, BoolQ, and PIQA. Zero-shot testing was performed on the OpenBookQA (OBQA) and ARC-Challenge tasks, which were not included in the initial training. Figure 10 As shown, on OpenBookQA, this embodiment (dark blue bar) achieved an accuracy of 77.2%, surpassing the standard LoRA's 75.4%, and on ARC-Challenge, this embodiment achieved an accuracy of 79.7%, also surpassing the standard LoRA's 78.7%. Figure 10 The orange area on the right (Task Set 2: Social Common Sense Tasks) indicates that the model should be trained only on a subset of tasks containing HellaS and WinoG. Testing should then be conducted on unseen SIQA tasks, such as... Figure 10 As shown, this embodiment (dark orange bars) achieved an accuracy of 74.3% on SIQA, which is superior to the standard LoRA's 73.2%. Among these, Figure 10 In the diagram, the gray diagonal bars represent the unadjusted zero-sample baseline, while the light-colored bars represent the control group with randomly assigned loops. Therefore, standard LoRA joint learning, by indiscriminately updating all attention heads of the model during fine-tuning, is prone to introducing noise or overfitting the model to shortcuts in the current training task, resulting in poor performance on similar new tasks. This embodiment uses precise clustering gating optimization to strictly focus parameter updates on necessary clustering loops. This precise parameter isolation mechanism ensures that the underlying inference logic learned by the model from the training subset can be effectively mapped and reused on other unseen tasks within the same cluster (i.e., shared paths), thereby achieving better forward transfer and generalization capabilities.

[0081] In some embodiments, this embodiment also verifies the beneficial effect of mitigating the catastrophic forgetting phenomenon of large language models in continuous learning scenarios. Specifically, this embodiment divides the commonsense reasoning task into two stages for continuous learning experiments. When the number of clusters is set to 2, the model is first trained using data from the first task cluster, and its performance is benchmarked on the test set of the first task cluster. Then, the model weights from the first stage of training are used to train the second task cluster. Finally, the test set of the first task cluster is used again to evaluate the model's accuracy decline on the old task. Experimental results show that standard LoRA fine-tuning updates parameters indiscriminately after learning a new task, resulting in severe overwriting of the feature representations of the old task, leading to a decrease in the average accuracy on the LLaMA3 model. In contrast, the method provided in this embodiment utilizes the structural sparsity of clustering loops to strictly limit parameter updates to the attention head region of the corresponding cluster when training a new task, effectively isolating the routing updates of specific clusters, thereby significantly reducing the model performance decline and achieving a near-zero forgetting continuous learning effect.

[0082] Please see Figure 11 This application embodiment also provides a training device 1100 for a multi-task model, which can implement the above-described training method for the multi-task model. The device includes: Acquisition unit 1110 is used to acquire training data for multiple tasks; The generation unit 1120 is used to generate an importance evaluation result of each attention point to the task based on the prediction results obtained by the multi-task model based on the training data for each task. The partitioning unit 1130 is used to partition multiple tasks based on all importance assessment results to obtain multiple task sets; The filtering unit 1140 is used to filter the corresponding target attention head from multiple attention heads based on the importance evaluation results corresponding to each task set; The construction unit 1150 is used to construct a corresponding mask matrix based on the target attention head corresponding to each task set. The mask value representation parameter of the target attention head in the mask matrix is ​​activated, and the mask value representation parameter of attention heads other than the target attention head in the mask matrix is ​​frozen. The computing unit 1160 is used to extract target training data of the target task set from the training data, and to perform gradient calculation based on the target training data through a multi-task model to obtain the first gradient of the multilayer perceptron module and the second gradient of each attention head. The update unit 1170 is used to update the global parameters of the multilayer perceptron module according to the first gradient, and to update the local parameters of the attention module according to the mask matrix corresponding to the target task set and the second gradient, so as to obtain the trained target multi-task model.

[0083] The specific implementation of the training device for this multi-task model is basically the same as the specific implementation of the training method for the multi-task model described above, and will not be repeated here.

[0084] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the training method of the aforementioned multi-task model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0085] Please see Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1201 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1202 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the training method of the multi-task model of the embodiments of this application. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0086] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method for the multi-task model described above.

[0087] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0088] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0089] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0092] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0093] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0095] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A training method for a multi-task model, characterized in that, The multi-task model includes an attention module and a multilayer perceptron module, the attention module including multiple attention heads, and the method includes: Acquire training data for multiple tasks; For each task, an importance assessment result for each attention head to the task is generated based on the prediction result obtained by the multi-task model based on the training data. Based on all the importance assessment results, the multiple tasks are divided into multiple task sets; Based on the importance assessment results corresponding to each of the task sets, the corresponding target attention head is obtained by filtering from the multiple attention heads; A corresponding mask matrix is ​​constructed based on the target attention head corresponding to each task set. The mask value representation parameter of the target attention head in the mask matrix is ​​activated, and the mask value representation parameter of attention heads other than the target attention head is frozen in the mask matrix. The target training data of the target task set is extracted from the training data, and gradient calculation is performed based on the target training data by the multi-task model to obtain the first gradient of the multilayer perceptron module and the second gradient of each attention head. The multilayer perceptron module is globally updated based on the first gradient, and the attention module is locally updated based on the mask matrix corresponding to the target task set and the second gradient, to obtain the trained target multi-task model.

2. The method according to claim 1, characterized in that, The prediction result includes a first loss value and a second loss value. For each task, based on the prediction result obtained by the multi-task model from the training data, an importance evaluation result for each attention head to the task is generated, including: A first loss value is obtained, which is calculated by the multi-task model based on the original model parameters and the training data corresponding to each task. Perform a masking operation on each attention head to obtain the masked updated model parameters; A second loss value is obtained, which is calculated by the multi-task model based on the mask to update the model parameters and the training data corresponding to each task. For each task, an importance assessment result for each attention head to the task is generated based on the first loss value and the second loss value.

3. The method according to claim 2, characterized in that, The importance assessment results include importance scores. Based on all the importance assessment results, the multiple tasks are divided into multiple task sets, including: Calculate the importance score of the attention head to the task based on the first loss value and the second loss value; For each of the tasks, a task feature embedding vector is generated based on all importance scores of the task. Calculate the distance between the task feature embedding vectors corresponding to each pair of tasks, and divide the multiple tasks according to the distance values ​​to obtain multiple task sets.

4. The method of claim 2, wherein, The step of selecting the corresponding target attention head from the multiple attention heads based on the importance evaluation results corresponding to each task set includes: For each task set, a corresponding clustering importance score is calculated based on the number of tasks contained in the task set and the importance evaluation results of all tasks in the task set. Based on the cluster importance scores, a preset number of target attention heads are selected from the multiple attention heads in descending order.

5. The method of claim 1, wherein, The step of constructing a corresponding mask matrix based on the target attention head corresponding to each task set includes: Obtain the size parameters of the attention module; Construct a corresponding mask matrix based on the target attention head and the size parameters for each task set.

6. The method of claim 1, wherein, The step of updating the local parameters of the attention module based on the mask matrix corresponding to the target task set and the second gradient to obtain the trained target multi-task model includes: The target mask gradient is obtained by multiplying the mask matrix corresponding to the target task set element by element with each of the second gradients. The attention module's local parameters are updated based on the target mask gradient to obtain the trained target multi-task model.

7. The method of claim 1, wherein, After updating the global parameters of the multilayer perceptron module according to the first gradient and updating the local parameters of the attention module according to the mask matrix corresponding to the target task set and the second gradient to obtain the trained target multi-task model, the method further includes: Obtain the task to be predicted, determine the set of target tasks that match the task to be predicted, and determine the second target mask matrix corresponding to the set of target tasks; The target multi-task model is invoked to generate the task prediction result of the task to be predicted based on the second target mask matrix.

8. A training apparatus of a multitask model, characterized by, The multi-task model includes an attention module and a multilayer perceptron module. The attention module includes multiple attention heads. The device includes: The acquisition unit is used to acquire training data for multiple tasks; The generation unit is configured to generate an importance evaluation result for each attention head to the task based on the prediction result obtained by the multi-task model based on the training data for each task. A partitioning unit is used to partition the multiple tasks based on all the importance evaluation results to obtain multiple task sets; A filtering unit is used to filter the multiple attention heads to obtain the corresponding target attention head based on the importance evaluation result corresponding to each task set; The construction unit is used to construct a corresponding mask matrix based on the target attention head corresponding to each task set. The mask value representation parameter of the target attention head in the mask matrix is ​​activated, and the mask value representation parameter of attention heads other than the target attention head is frozen in the mask matrix. The computing unit is used to extract target training data of the target task set from the training data, and to perform gradient calculation based on the target training data through the multi-task model to obtain the first gradient of the multilayer perceptron module and the second gradient of each attention head. The update unit is used to update the global parameters of the multilayer perceptron module according to the first gradient, and to update the local parameters of the attention module according to the mask matrix corresponding to the target task set and the second gradient, so as to obtain the trained target multi-task model.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the training method of the multi-task model according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. When the computer program is executed by the processor, it implements the training method of the multi-task model according to any one of claims 1 to 7.

Images