Multi-task learning resource optimization method based on lottery assumption

Abstract

This invention provides a multi-task learning resource optimization method based on the lottery hypothesis, belonging to the field of multi-task learning and neural network optimization technology. This method employs a sparse expert hybrid approach based on the lottery hypothesis to effectively prune neural network model parameters, thereby significantly reducing the computational cost and storage overhead of the neural network model. The sparse expert network structure retains high sensitivity to key task information. The introduction of a softmax-based router and mask matrix allows each task to select and activate suitable expert network components based on the features of its input data. Cross-training and iterative amplitude pruning optimize the omnipotent expert matrix, further enhancing the model's generalization ability in handling complex tasks and different data distributions. Task-specific layers independently process and optimize the features of each task, reducing mutual interference between tasks and improving the accuracy of feature extraction and the efficiency of task processing.

Description

Technical Field

[0001] This invention relates to the field of multi-task learning and neural network optimization technology, and in particular to a multi-task learning resource optimization method based on lottery assumptions. Background Technology

[0002] The lottery hypothesis is a pruning method used in deep learning for compressing neural network models. It posits that complex deep neural networks possess an optimized sparse subnetwork structure (called winning tickets) that can be applied to model compression. Compared to the original network, the sparse subnetwork has significantly fewer parameters and lower complexity, while maintaining comparable inference accuracy, thus reducing computational cost and storage overhead.

[0003] Currently, multi-task learning is widely used in scenarios requiring the simultaneous processing of multiple related or unrelated tasks, and has become an important technical means to improve model performance and optimize the utilization of computing resources. The core idea of ​​multi-task learning is to improve learning efficiency and enhance the model's generalization ability by sharing relevant information among tasks. Traditional multi-task learning methods mainly include hard parameter sharing, soft parameter sharing, expert sharing, and sparse sharing. These methods achieve information sharing and computing resource optimization among tasks to a certain extent, but when dealing with tasks with significant differences in relevance or complexity, they still struggle to effectively balance resource allocation and feature extraction requirements among tasks. These methods may lead to wasted computing resources and interference between tasks, thus affecting the overall performance of the model.

[0004] In recent years, Multi-gate Mixture-of-Experts (MMoE) models have gained widespread attention as an improved multi-task learning method. MMoE models introduce multiple expert networks and independent task gating networks, allowing each task to select and combine different expert outputs according to its own needs. Specifically, each task in an MMoE model has an independent gating network, which determines the weights of each expert based on the task's input features, thereby dynamically selecting and activating experts suitable for the current task. This mechanism enhances the model's flexibility to a certain extent and avoids direct conflicts and information interference between tasks.

[0005] While Multi-Gate Hybrid Expert Models (MMoEs) achieve task-specific expert selection by introducing independent task-gated networks, they still face challenges such as high computational resource consumption, redundant computation, and lack of flexibility. When processing multiple tasks, MMoE models need to compute the outputs of multiple experts simultaneously for each task, leading to resource waste and increased computational overhead. Furthermore, fixed expert and task-gated network structures struggle to adapt to task variations and complex task requirements, limiting the model's dynamic adjustment capabilities. Especially when task relevance is low, existing MMoE models still fall short in feature sharing between tasks, failing to effectively utilize shared information and resulting in suboptimal overall efficiency and performance. Therefore, a mechanism capable of dynamically adjusting resource allocation and expert selection is urgently needed to improve the flexibility and resource utilization efficiency of multi-task learning. This mechanism should flexibly select and activate different parts of the expert network according to task requirements, reduce computational resource waste, and enhance information sharing between tasks, thereby improving the overall performance and efficiency of multi-task learning. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a multi-task learning resource optimization method based on the lottery hypothesis. This method uses mask selection and the lottery hypothesis principle to efficiently activate and optimize specific parts of the omnipotent expert layer, making the network more sparse and achieving task-specific high-level feature extraction and computational resource optimization.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a multi-task learning resource optimization method based on the lottery hypothesis, comprising the following steps:

[0008] Step 1: Obtain the original datasets for each task and divide them into training and testing sets to ensure that each task has an independent dataset for testing and evaluating the neural network model;

[0009] Select the original datasets for task a and task b, and divide them into a training set for task a. and test set and the training set for task b and test set

[0010] The data in the original dataset includes images, text, audio, and video;

[0011] Step 2: Use an embedding layer to convert the input data of task a and task b into low-dimensional continuous vector representations, i.e., generate embedding vectors for the input data of task a and task b; use the softmax function as a router to calculate the routing probability data of task a and task b based on the generated embedding vectors; initialize the common mask matrix of task a and task b, and randomly initialize the omnipotent expert matrix, specifically as follows:

[0012] S1: Use an embedding layer to convert the input data of each task into a low-dimensional continuous vector representation; convert the input data X of task a... (a) And the input data X for task b (b) Each vector is converted into a low-dimensional continuous vector representation through an embedding layer. The specific conversion formula is as follows:

[0013] E (a) =Embed(X) (a) )

[0014] E (b) =Embed(X) (b) )

[0015] Among them, E (a) Input data X (a) The embedding vector, E (b) Input data X (b) The embedding vector, where Embed is the embedding layer;

[0016] S2: Design the router to initialize the routing probability vectors for tasks a and b; use the softmax function as the router, based on the embedding vector E. (a) and embedding vector E (b) Initialize the routing probability vectors of task a and task b and in These are the n expert coefficients for task a. These are the n expert coefficients for task b, calculated using the following formula:

[0017] α a =soft max x(E) (a) )

[0018] α b =softmax(E (b) )

[0019] S3: Initialize the mask matrices M1, M2, M3...M common to tasks a and b. n Each element of the mask matrix is ​​initialized to 1; an all-around expert matrix is ​​randomly initialized for subsequent training, and the mask matrices M1, M2, M3...M are used as the training vectors. n Select and activate specific parameters within the omnipotent expert matrix;

[0020] Step 3: Randomly initialize a set of neural network model parameters E; jointly train the initialized n mask matrices using the training sets of tasks a and b; after training on the training set, use iterative amplitude pruning based on the lottery assumption principle to remove the parameters with the last p% absolute value in the neural network model parameters E, and then update the mask matrix; use the updated mask matrix to select and activate specific parameters in the omnipotent expert matrix, including the following steps:

[0021] Step 3.1: Use the training sets of Task a and Task b and Jointly train the initial n mask matrices; combine the training sets of task a and task b. and The data is divided into mini-batches, and the mini-batches of task a and task b are forward-propagated separately to obtain the loss functions L for task a and task b. a and L b Then, the loss functions L for tasks a and b are... a and L b Adding them together yields the loss function L. total The specific formula is as follows:

[0022] L total =L a +L b

[0023] Backpropagation of the loss function L using the gradient descent algorithm total The loss function L total Minimize and update the neural network model parameters E;

[0024] Step 3.2: After training the entire original dataset as in Step 3.1, perform iterative amplitude pruning based on the lottery hypothesis. Set the pruning ratio p, and prune p% of the parameters after removing the absolute value of the neural network model parameters. Then, use the mask matrices M1, M2, M3...M n The corresponding mask is set to 0; the mini-batch data is re-divided for the training sets of task a and task b. and Retrain, performing iterative amplitude pruning after each training iteration until a set sparsity level is reached, then stop training. The mask matrices are M1, M2, M3...M n Training completed;

[0025] Step 4: Perform cross-training using the training sets of Task a and Task b. Update the parameters of the omnipotent expert matrix after each training round until the parameters of the omnipotent expert matrix converge. Stop training to obtain the final omnipotent expert matrix. This includes the following steps:

[0026] Step 4.1: Use the training sets of Task a and Task b and Perform cross-training; combine the training sets of task a and task b. and Divide the data into multiple mini-batches, perform forward propagation on a mini-batch of data from task a, and derive the loss function L. a Then, backpropagation is performed, and gradient descent is used to adjust the loss function L. a Minimize the loss function L, at which point the parameters of the omnipotent expert matrix are updated accordingly; then, use a mini-batch of data from task b for forward propagation to obtain the loss function L. b ', Use gradient descent to reduce the loss function L b Minimize, backpropagation minimizes the loss function L b Then update the parameters of the all-around expert matrix;

[0027] Step 4.2: Next, following the cross-training method in Step 4.1, train the mini-batch data of Task a and Task b. Update the parameters of the omnipotent expert matrix after each training session until the parameters converge. Stop training and obtain the final omnipotent expert matrix E. all Training complete;

[0028] Step 5: Use the test set of task a and the test set for task b The data in the file will initialize the probability vectors α of task a and task b. a and α b The result is multiplied by the trained sparse mask matrix and the omnipotent expert matrix, and finally the output of the final task is generated through a tower structure. The result depends on the specific task.

[0029] Using the routing probability vectors α of task a and task b a and α b As input to train the neural network model, each element in the vector is compared with the trained sparse mask matrix M'1, M'2, M'3...M'. n And the all-around expert matrix E all Multiplication, i.e., activating only specific parameters of the omnipotent expert matrix, and then summing the results, i.e., activating only the sparse mask matrices M'1, M'2, M'3...M' n The omnipotent expert matrix parameters corresponding to the positions with a median value of 1 are used to obtain the output result Y of the entire neural network model. a and Y b :

[0030]

[0031] Finally, Y a and Yb By using the pyramid structure, the relevant features of task a and task b are learned.

[0032] The beneficial effects of adopting the above technical solution are as follows: The multi-task learning resource optimization method based on the lottery hypothesis provided by this invention has the following advantages compared with the prior art:

[0033] (1) The sparse expert hybrid method with lottery assumption is adopted to effectively prune the parameters of the neural network model, thereby greatly reducing the computational cost and storage overhead of the neural network model; the sparse expert network structure retains high sensitivity to key information of the task, while improving the efficiency of the model while ensuring inference accuracy.

[0034] (2) By introducing a softmax-based router and mask matrix, dynamic resource allocation is realized, which enables each task to select and activate the appropriate expert network part according to the characteristics of its input data. This mechanism enhances the flexibility of the multi-task learning model in information sharing between different tasks and effectively reduces the waste of computing resources and redundant computation.

[0035] (3) By optimizing the all-around expert matrix through cross-training and iterative amplitude pruning, the generalization ability of the model in handling complex tasks and different data distributions is further improved.

[0036] (4) The task-specific tower layers independently process and optimize the features of each task, reducing mutual interference between tasks and improving the accuracy of feature extraction and the efficiency of task processing. Attached Figure Description

[0037] Figure 1 A flowchart of a multi-task learning resource optimization method based on lottery assumptions is provided in this embodiment of the invention;

[0038] Figure 2 This invention provides an architecture diagram for a multi-task learning resource optimization method based on lottery assumptions. Detailed Implementation

[0039] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0040] In this embodiment, a multi-task learning resource optimization method based on lottery assumptions is described, such as... Figure 1-2 As shown, it includes the following steps:

[0041] Step 1: Obtain the original datasets for each task and divide them into training and testing sets to ensure that each task has an independent dataset for testing and evaluating the neural network model;

[0042] Select the original datasets for task a and task b, and divide them into a training set for task a. and test set and the training set for task b and test set

[0043] The data in the original dataset includes various forms such as images, text, audio, and video;

[0044] In this embodiment, task a is an image detection task, and task b is an object classification task. The input data for both task a and task b are images.

[0045] Step 2: Use an embedding layer to convert the input data of task a and task b into low-dimensional continuous vector representations, i.e., generate embedding vectors for the input data of task a and task b; use the softmax function as a router to calculate the routing probability data of task a and task b based on the generated embedding vectors; initialize the common mask matrix of task a and task b (set all to 1), and randomly initialize the omnipotent expert matrix, specifically as follows:

[0046] S1: Use an embedding layer to convert the input data of each task into a low-dimensional continuous vector representation; convert the input data X of task a... (a) And the input data X for task b (b) Each vector is converted into a low-dimensional continuous vector representation through an embedding layer. The specific conversion formula is as follows:

[0047] E (a) =Embed(X) (a) )

[0048] E (b) =Embed(X) (b) )

[0049] Among them, E (a) Input data X (a) The embedding vector, E (b) Input data X (b) The embedding vector, where Embed is the embedding layer;

[0050] S2: Design the router to initialize the routing probability vectors for tasks a and b; use the softmax function as the router, based on the embedding vector E. (a) and embedding vector E (b) Initialize the routing probability vectors of task a and task b and in These are the n expert coefficients for task a. These are the n expert coefficients for task b, calculated using the following formula:

[0051] α a =softmax(E (a) )

[0052] α b =softmax(E (b) )

[0053] S3: Initialize the mask matrices M1, M2, M3...M common to tasks a and b. n (The number of mask matrices initialized corresponds to the number of elements in the routing probability vector), with each element of the mask matrix initialized to 1; a jack-of-all-trades expert matrix is ​​randomly initialized for subsequent training, and the mask matrices M1, M2, M3...M are used as the basis for the training. n Select and activate specific parameters within the omnipotent expert matrix;

[0054] Step 3: Randomly initialize a set of neural network model parameters E; jointly train the initialized n mask matrices using the training sets of tasks a and b. After training on the training set, based on the lottery hypothesis principle, use iterative amplitude pruning to remove the p% of parameters (unimportant parameters) after their absolute values ​​in the neural network model parameters E, and then update the mask matrices; after A rounds of iterative amplitude pruning, the mask matrices will gradually become sparse after reaching the target sparsity, retaining the parameters most important to tasks a and b, including the following steps:

[0055] Step 3.1: Use the training sets of Task a and Task b and Jointly train the initial n mask matrices; combine the training sets of task a and task b. and The data is divided into multiple mini-batches. Each mini-batch from task a and each mini-batch from task b are then forward-propagated to obtain the loss function L for tasks a and b. a and L b Then, the loss functions L for tasks a and b are... a and L b Adding them together yields the loss function L. total The specific formula is as follows:

[0056] L total =L a +L b

[0057] Backpropagation of the loss function L using the gradient descent algorithm total The loss function L total Minimize and update the neural network model parameters E;

[0058] In this embodiment, the cross-entropy function is used to calculate the loss function;

[0059] Step 3.2: After training the entire original dataset (epoch) as in Step 3.1, perform iterative amplitude pruning based on the lottery hypothesis. Set an appropriate pruning ratio p, and prune p% of the parameters after removing the absolute value of the neural network model parameters. Then, use the mask matrices M1, M2, M3...M n The corresponding mask is set to 0; the mini-batch data is re-divided for the training sets of task a and task b. and Retrain, performing iterative amplitude pruning after each training iteration until a set sparsity level is reached, then stop training. The mask matrices are M1, M2, M3...M n Training completed;

[0060] Step 4: Perform cross-training using the training sets of Task a and Task b. Update the parameters of the omnipotent expert matrix after each training round until the parameters of the omnipotent expert matrix converge. Stop training to obtain the final omnipotent expert matrix. This includes the following steps:

[0061] Step 4.1: Use the training sets of Task a and Task b and Perform cross-training; combine the training sets of task a and task b. and The data is divided into multiple mini-batches. One mini-batch of data from task a is then forward-propagated to derive the loss function L. a Then, backpropagation is performed, and gradient descent is used to adjust the loss function L. a Minimize the loss function L, at which point the parameters of the omnipotent expert matrix are updated accordingly; then, use a mini-batch of data from task b for forward propagation to obtain the loss function L. b ', Use gradient descent to reduce the loss function L b Minimize, backpropagation minimizes the loss function L b Then update the parameters of the all-around expert matrix;

[0062] Step 4.2: Next, following the cross-training method in Step 4.1, train the mini-batch data of Task a and Task b. Update the parameters of the omnipotent expert matrix after each training session until the parameters converge. Stop training and obtain the final omnipotent expert matrix E. all Training complete;

[0063] Step 5: Use the test set of task a and the test set for task b The data in the file will initialize the probability vectors α of task a and task b. a and α bMultiplying the trained sparse mask matrix and the omnipotent expert matrix, i.e. only activating specific parameters of the omnipotent expert matrix, and finally generating the final task output through a tower structure, the result depends on the specific task.

[0064] Using the routing probability vectors α of task a and task b a and α b As input to train the neural network model, each element in the vector is compared with the trained sparse mask matrix M'1, M'2, M'3...M'. n And the all-around expert matrix E all Multiply them, then add the results, which means only activating the sparse mask matrices M'1, M'2, M'3...M' n The omnipotent expert matrix parameters corresponding to the positions with a median value of 1 are used to obtain the output result Y of the entire neural network model. a and Y b :

[0065]

[0066] Finally, Y a and Y b By using the pyramid structure, we can learn the relevant features of task a and task b.

[0067] A hierarchical structure is a network structure specialized for each task, allowing each task to learn specific features relevant to itself to adapt to the needs of a particular task.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multi-task learning resource optimization method based on lottery assumptions, characterized in that: Includes the following steps: Step 1: Obtain the original datasets for each task and divide them into training and testing sets to ensure that each task has an independent dataset for testing and evaluating the neural network model; Select the original datasets for task a and task b, and divide them into a training set for task a. and test set and the training set for task b and test set ; The data in the original dataset includes images, text, audio, and video; task a is an image detection task, and task b is an object classification task. The input data for both task a and task b are images. Step 2: Use an embedding layer to convert the input data of task a and task b into low-dimensional continuous vector representations, i.e., generate embedding vectors for the input data of task a and task b; use the softmax function as a router to calculate the routing probability data of task a and task b based on the generated embedding vectors; initialize the mask matrix common to task a and task b, and randomly initialize the omnipotent expert matrix. Step 3: Randomly initialize a set of neural network model parameters E; jointly train the initialized n mask matrices using the training sets of task a and task b; after the training set is completed, use iterative amplitude pruning based on the lottery hypothesis to remove the absolute values ​​from the neural network model parameters E. The parameters are then updated, and the mask matrix is ​​used to select and activate specific parameters in the omnipotent expert matrix. Step 4: Perform cross-training using the training sets of Task a and Task b. Update the parameters of the omnipotent expert matrix after each round of training until the parameters of the omnipotent expert matrix converge. Stop training and obtain the final omnipotent expert matrix. Step 5: Use the test set of task a and the test set for task b The data in the file will initialize the probability vectors of task a and task b. and The result is multiplied by the trained sparse mask matrix and the omnipotent expert matrix, and finally the output of the final task is generated through a tower structure. The result depends on the specific task. Using the routing probability vectors of task a and task b and As input to train the neural network model, each element in the vector is compared with the trained sparse mask matrix. and the all-around expert matrix Multiplication, which activates only a specific subset of parameters in the omnipotent expert matrix, and then summing the results, activates only the sparse mask matrix. The omnipotent expert matrix parameters corresponding to the positions with a median value of 1 are used to obtain the output of the entire neural network model. and : ； ； in, Tasks n expert coefficients, Tasks n expert coefficients; Finally, and By using the pyramid structure, the relevant features of task a and task b are learned.

2. The multi-task learning resource optimization method based on the lottery hypothesis according to claim 1, characterized in that: Step 2 is as follows: S1: Use an embedding layer to convert the input data of each task into a low-dimensional continuous vector representation; convert the input data of task a... Input data for task b Each vector is converted into a low-dimensional continuous vector representation through an embedding layer. S2: Design the router to initialize the routing probability vectors for tasks a and b; use the softmax function as the router, based on the embedding vectors... and embedding vector Initialize the routing probability vectors of task a and task b and ,in Tasks n expert coefficients, Tasks n expert coefficients; S3: Initialize the mask matrix shared by tasks a and b Each element of the mask matrix is ​​initialized to 1; an omnipotent expert matrix is ​​randomly initialized for subsequent training, and the results are obtained through the mask matrix. Select and activate specific parameters within the omnipotent expert matrix.

3. The multi-task learning resource optimization method based on the lottery hypothesis according to claim 2, characterized in that: The input data of task a Input data for task b The specific transformation formulas for converting the vectors into low-dimensional continuous vector representations through embedding layers are as follows: ； ； in, For input data Embedded vector, For input data The embedding vector, where Embed is the embedding layer.

4. The multi-task learning resource optimization method based on the lottery hypothesis according to claim 3, characterized in that: The softmax function is used as a router, based on the embedding vector. and embedding vector Initialize the routing probability vectors of task a and task b and The specific calculation formula is as follows: ； 。 5. The multi-task learning resource optimization method based on the lottery hypothesis according to claim 4, characterized in that: Step 3 includes the following steps: Step 3.1: Use the training sets of Task a and Task b and Jointly train the initial n mask matrices; combine the training sets of task a and task b. and The data is divided into mini-batches, and the mini-batches of task a and task b are forward-propagated separately to obtain the loss functions for task a and task b. and Then the loss functions for task a and task b are... and Adding them together gives the loss function The specific formula is as follows: ； Backpropagation of the loss function using gradient descent algorithm. loss function Minimize and update the neural network model parameters E; Step 3.2: After training the entire original dataset as described in Step 3.1, perform iterative amplitude pruning based on the principles of the lottery hypothesis, and set the pruning ratio. After removing the absolute values ​​of the neural network model parameters The parameters will be the mask matrix. The corresponding mask is set to 0; the mini-batch data is re-divided for the training sets of task a and task b. and Retrain, performing iterative amplitude pruning after each training iteration until the set sparsity is reached, then stop training and set the mask matrix. Training complete.

6. The multi-task learning resource optimization method based on the lottery hypothesis according to claim 5, characterized in that: Step 4 includes the following steps: Step 4.1: Use the training sets of Task a and Task b and Perform cross-training; combine the training sets of task a and task b. and Divide the data into multiple mini-batches, perform forward propagation on a mini-batch of data from task a, and derive the loss function. Then, backpropagation is performed, and gradient descent is used to adjust the loss function. Minimize the loss function, at which point the parameters of the omnipotent expert matrix are also updated; then, use a mini-batch of data from task b for forward propagation to derive the loss function. Using gradient descent to reduce the loss function Minimize, backpropagation minimizes the loss function Then update the parameters of the all-around expert matrix; Step 4.2: Next, following the cross-training method in Step 4.1, train the mini-batch data of Task a and Task b. Update the parameters of the omnipotent expert matrix after each training iteration until the parameters converge. Stop training to obtain the final omnipotent expert matrix. Training complete.

Images