Multi-task network training method and apparatus, and computing device cluster

Abstract

A multi-task network training method. A multi-task network comprises a backbone network. The backbone network comprises a network layer having a weight matrix, wherein the network layer has a one-hot mask matrix, and the one-hot mask matrix comprises T (T≥2) submatrices, one submatrix being related to one task, and the submatrices being used for masking parameters in the network layer. The method comprises: acquiring the gradients of T tasks backpropagated to a network layer; on the basis of the gradients backpropagated to the network layer, adjusting the values of elements in a one-hot mask matrix associated with the network layer; and on the basis of the adjusted one-hot mask matrix and the gradients backpropagated to the network layer, updating parameters of a backbone network. Thus, a mapping relationship between tasks and channels can be constructed in a trained backbone network, such that different tasks can use different channels of a network layer in the backbone network during training and inference, thereby resolving task conflicts at a channel level.

Description

Multi-task network training method and device and computing device cluster

[0001] This application claims priority to the Chinese patent application No. 2024104586415, filed on April 15, 2024, entitled "Multi-task network training method and device and computing device cluster", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a multi-task network training method, device and computing device cluster. BACKGROUND

[0003] Multi-task learning (MTL) is a subfield of machine learning, which generally refers to learning multiple tasks simultaneously through a shared model. Multi-task learning can bring many benefits, such as improving data utilization efficiency, or reducing overfitting through shared features, or improving learning speed through knowledge sharing between tasks. However, in many cases, due to conflicts between tasks, it is very difficult for the model to learn the knowledge of multiple tasks simultaneously. Such conflicts between tasks are called negative transfer or destructive interference in the field of multi-task learning. Since the shared module of the model binds the parameters of different tasks, improving the performance of the model on one or some tasks may be harmful to the performance of the model on other tasks. Therefore, how to solve the conflicts between tasks in multi-task learning is a technical problem that needs to be solved at present. SUMMARY

[0004] The present application provides a multi-task network training method, device, computing device cluster, computer storage medium and computer product, which can solve the conflicts between tasks in multi-task learning.

[0005] In a first aspect, the present application provides a multi-task network training method, wherein the trained multi-task network comprises a backbone network, and the backbone network comprises a network layer with a weight matrix, and the network layer has a one-hot mask matrix, and the one-hot mask matrix comprises T (T≥2) sub-matrices, and each of the T sub-matrices is related to a task, and the sub-matrices are used to mask the parameters in the network layer. The method comprises: obtaining gradients of T tasks returned to the network layer; adjusting values of elements in the one-hot mask matrix associated with the network layer based on the gradients returned to the network layer; and updating parameters of the backbone network based on the adjusted one-hot mask matrix and the gradients returned to the network layer.

[0006] Thus, since the sub-matrices in the one-hot mask matrix can be used to mask at least part of the parameters in the network layer, after the adjustment of the values of the elements in the one-hot mask matrix is completed, different sub-matrices can mask different parameters in the network layer. Each sub-matrix in the one-hot mask matrix is related to a task, so that when different tasks are inferred, different parameters in the network layer can be used to train or infer different tasks, which establishes the mapping relationship between the tasks and the channels, so that different tasks can use different channels of the network layer in the backbone network during training and inference, thereby realizing the solution of task conflict at the channel level in multi-task learning. For example, the channel here can refer to the output channel in the network layer. In addition, the one-hot mask matrix can be a form of one-hot encoding, which is a multi-dimensional array or matrix, and the elements in it are composed of 0 or 1. When the value of an element in the one-hot mask matrix is 0, it means that the parameters in the network layer related to the element are masked (i.e., not used); when the value of an element in the one-hot mask matrix is 1, it means that the parameters in the network layer related to the element are not masked (i.e., used).

[0007] In a possible implementation, based on the gradient backpropagated to the network layer, the values of the elements in the one-hot mask matrix associated with the network layer are adjusted, including: based on the gradient backpropagated to each channel in the network layer from T tasks, respectively determining the weight of each channel in the network layer allocated to each task of the T tasks; based on the weight of each channel in the network layer allocated to each task of the T tasks, adjusting the values of the elements in the one-hot mask matrix associated with the network layer. In this way, by calculating the weight of each channel allocated to different tasks through the gradient backpropagated to the channel by each task and the parameters of each channel itself, the mapping relationship between the tasks and the channels can be established through the calculated weight.

[0008] In a possible implementation, based on the gradients of the T tasks back to each channel in the network layer, weights of each channel in the network layer allocated to each of the T tasks are determined respectively, including: based on parameters of any one channel in the network layer and the gradient of the first task back to the any one channel, a loss disturbance degree of the any one channel to the first task is calculated, the first task being any one of the T tasks; based on the gradient of the first task back to the any one channel and the gradient of other tasks than the first task of the T tasks back to the any one channel, a gradient similarity of the first task and the other tasks on the any one channel is calculated; and based on the loss disturbance degree and the gradient similarity, a weight of the any one channel allocated to the first task is calculated. In this way, the importance of the channel to the loss optimization of the task can be intuitively seen through the loss disturbance degree. For example, when the loss disturbance degree of a channel to a task is very large, it indicates that the channel is very important to the task, and therefore the channel should be allocated to the corresponding task. In addition, the conflict degree between different tasks on a channel can be observed through the gradient similarity between different tasks on the channel. For example, when the gradient similarity of two tasks on a channel is very large, it indicates that the conflict of the two tasks on the channel is very small, and therefore the two tasks can share a channel. Further, the loss disturbance degree and the gradient similarity are combined to calculate the weight, so that the two factors can be considered comprehensively, so that the calculated weight can take both into account, so that the allocated channel can not only alleviate the task conflict but also ensure a smaller loss.

[0009] In a possible implementation, based on the weights of each channel in the network layer allocated to each of the T tasks, values of elements in the one-hot mask matrix associated with the network layer are adjusted, including: in a case where the weight of any one channel in the network layer allocated to the first task is less than a first threshold, a value of an element related to the any one channel in the first sub-matrix is set to 0, the first task being any one of the T tasks and the first sub-matrix being a matrix related to the first task in the one-hot mask matrix associated with the network layer; and in a case where the weight of the any one channel in the network layer allocated to the first task is greater than or equal to the first threshold, the value of the element related to the any one channel in the first sub-matrix is set to 1. Exemplarily, the first threshold can be but is not limited to a quantile of the weight of each channel allocated to each task. In this way, by comparing the weight with the threshold, the adjustment of the elements in the one-hot mask matrix can be realized, so as to realize the construction of the mapping relationship between the tasks and the channels. Exemplarily, setting the value of the element to 0 can be understood as adjusting the value of the element from 1 to 0, or keeping the value of the element as 0; and setting the value of the element to 1 can be understood as adjusting the value of the element from 0 to 1, or keeping the value of the element as 1.

[0010] In a possible implementation, the parameters of the backbone network are updated based on the adjusted one-hot mask matrix associated with the network layer and the gradient backpropagated to the network layer, including: calculating a new gradient of the network layer based on the adjusted one-hot mask matrix associated with the network layer and the gradient backpropagated to the network layer; and updating the parameters of the backbone network based on the new gradient of the network layer. In this way, the one-hot mask matrix can be used to update the gradient, so that the new gradient obtained is related to the one-hot mask matrix, and then the updated parameters in the network layer can be associated with the one-hot mask matrix to implement the construction of the mapping relationship between the tasks and the channels.

[0011] In a possible implementation, the new gradient of the network layer is calculated based on the adjusted one-hot mask matrix associated with the network layer and the gradient backpropagated to the network layer, including: grouping the channels in the network layer based on the adjusted one-hot mask matrix associated with the network layer to obtain N (N≥2) channel groups, where different channel groups process different sets of tasks; using a gradient modulation algorithm to calculate the gradient on the channels included in each of the N channel groups to obtain a weighted weight of each channel group; and calculating the new gradient of the network layer based on the weighted weight of each channel group and the gradient backpropagated to the network layer. In this way, a reasonable channel grouping can be learned in a heuristic manner without searching in a large space, so that the channel grouping can be implemented and the channel grouping efficiency is improved. In addition, this channel grouping manner can also make the tasks in each channel group related, so that the conflict between the tasks can be alleviated. In addition, the gradient modulation algorithm divides the scope of the basic gradient modulation algorithm to the subnetwork parameters shared in each channel group, which can effectively handle the gradient conflict problem when different tasks use a multi-task model subnetwork.

[0012] In a possible implementation, the one-hot mask matrix is used for bit multiplication with the parameters of the network layer. For example, the one-hot mask matrix can be used for bit multiplication with the weight matrix of the network layer, and the mask matrix of the output channel included in the one-hot mask matrix can be used for bit multiplication with the bias term of the network layer. The bit multiplication, also known as element-wise multiplication, refers to multiplying the elements at the same position in different matrices.

[0013] In a possible implementation, before the gradients of the T tasks are obtained and fed back to the network layer in the backbone network, the method further includes: performing a logical OR operation on each sub-matrix in the one-hot mask matrix associated with the network layer to obtain a new sub-matrix associated with each of the T tasks, and performing processing on the tasks in the T tasks based on the new sub-matrix. For example, the logical OR operation on two sub-matrices in the one-hot mask matrix can be understood as follows: if the element at the same position in the two sub-matrices has a value of 1, the element at the position is set to 1; if the element at the same position in the two sub-matrices has a value of 0, the element at the position is set to 0.

[0014] In a possible implementation, the network layer with the weight matrix is multiple, and each network layer with the weight matrix includes a shared channel, and at least one network layer with the weight matrix further includes a non-shared channel; and the i th non-shared channel in the k th network layer with the weight matrix is connected to the shared channel and other channels containing the i th non-shared channel in the (k-1) th network layer with the weight matrix.

[0015] In a possible implementation, the initial value of the element in the one-hot mask matrix is 1.

[0016] In a possible implementation, after the training of the multi-task network is completed, the method further includes: obtaining user data; and performing inference on the user data by using the multi-task network; wherein during loading of the multi-task network, each one-hot mask matrix is multiplied by the parameters of the corresponding network layer, and the result obtained by the multiplication is used to perform parallel inference on each task in the inference; and the inference result of each task in the inference is output. In this way, during the loading process, the one-hot mask matrix is multiplied by the parameters of the corresponding network layer to update the parameters (such as weights and / or bias terms) in the network layer, so that the one-hot mask matrix can be directly discarded in the subsequent inference, and the multi-task learning network can be directly used for inference, so as to avoid introducing additional overheads caused by the one-hot mask matrix.

[0017] In a second aspect, the present application provides a multi-task network training apparatus. The trained multi-task network comprises a backbone network, and the backbone network comprises a network layer having a weight matrix, and the network layer has a one-hot mask matrix, and the one-hot mask matrix comprises T (T≥2) sub-matrices, and each of the T sub-matrices is associated with a task, and the sub-matrices are used to mask parameters in the network layer. The multi-task network training apparatus comprises an obtaining module and a processing module. The obtaining module is configured to obtain gradients of the T tasks returned to the network layer. The processing module is configured to adjust values of elements in the one-hot mask matrix associated with the network layer based on the gradients returned to the network layer, and update parameters of the backbone network based on the adjusted one-hot mask matrix and the gradients returned to the network layer.

[0018] In a possible implementation, when adjusting the values of the elements in the one-hot mask matrix associated with the network layer based on the gradients returned to the network layer, the processing module is specifically configured to determine weights of each channel in the network layer assigned to each of the T tasks based on the gradients of the T tasks returned to each channel in the network layer, respectively, and adjust the values of the elements in the one-hot mask matrix associated with the network layer based on the weights of each channel in the network layer assigned to each of the T tasks.

[0019] In a possible implementation, when determining the weights of each channel in the network layer assigned to each of the T tasks based on the gradients of the T tasks returned to each channel in the network layer, respectively, the processing module is specifically configured to calculate a loss disturbance degree of an arbitrary channel in the network layer to a first task based on parameters of the arbitrary channel and the gradient of the first task returned to the arbitrary channel, the first task being any one of the T tasks, calculate a gradient similarity of the first task and other tasks on the arbitrary channel based on the gradient of the first task returned to the arbitrary channel and the gradients of the other tasks returned to the arbitrary channel, and calculate the weight of the arbitrary channel assigned to the first task based on the loss disturbance degree and the gradient similarity.

[0020] In a possible implementation, when adjusting the values of the elements in the one-hot mask matrix associated with the network layer based on the weights of each channel in the network layer assigned to each of the T tasks, the processing module is specifically configured to set a value of an element in a first sub-matrix associated with an arbitrary channel to 0 in a case where the weight of the arbitrary channel in the network layer assigned to a first task is less than a first threshold, the first task being any one of the T tasks, and the first sub-matrix being a matrix in the one-hot mask matrix associated with the first task, and set the value of the element in the first sub-matrix associated with the arbitrary channel to 1 in a case where the weight of the arbitrary channel in the network layer assigned to the first task is greater than or equal to the first threshold.

[0021] In a possible implementation, when updating the parameters of the backbone network based on the adjusted one-hot mask matrix associated with the network layer and the gradient backpropagated to the network layer, the processing module is specifically configured to: calculate a new gradient of the network layer based on the adjusted one-hot mask matrix associated with the network layer and the gradient backpropagated to the network layer; and update the parameters of the backbone network based on the new gradient of the network layer.

[0022] In a possible implementation, when calculating the new gradient of the network layer based on the adjusted one-hot mask matrix associated with the network layer and the gradient backpropagated to the network layer, the processing module is configured to: group the channels in the network layer based on the adjusted one-hot mask matrix associated with the network layer to obtain N (N≥2) channel groups, where different channel groups process different sets of tasks; and calculate the gradient on each channel included in each of the N channel groups by using a gradient modulation algorithm to obtain a weighted weight of each channel group, and calculate the new gradient of the network layer based on the weighted weight of each channel group and the gradient backpropagated to the network layer.

[0023] In a possible implementation, the one-hot mask matrix is used for bit multiplication with the parameters of the network layer.

[0024] In a possible implementation, before the gradient of the T tasks backpropagated to the network layer in the backbone network is acquired by the acquisition module, the processing module is further configured to: take a logical OR of each sub-matrix in the one-hot mask matrix associated with the network layer as a new sub-matrix associated with each of the T tasks, and process the tasks in the T tasks based on the new sub-matrix.

[0025] In a possible implementation, there are a plurality of network layers each having a weight matrix, and each of the network layers each having a weight matrix includes: shared channels, and at least one of the network layers each having a weight matrix further includes: non-shared channels; and the i th non-shared channel in the k th network layer each having a weight matrix is connected with the shared channels in the (k-1) th network layer each having a weight matrix and other channels including the i th non-shared channel.

[0026] In a possible implementation, the initial value of an element in the one-hot mask matrix is 1.

[0027] In a possible implementation, after the training of the multi-task network is completed, the obtaining module 901 is further configured to obtain user data. The processing module 902 is further configured to perform inference on the user data by using the multi-task network; wherein during loading of the multi-task network, each one-hot mask matrix is multiplied with parameters of a corresponding network layer, and during the inference, the result obtained by the multiplication is used to perform parallel inference on each task in this inference; and the inference result of each task in this inference is output.

[0028] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium comprising computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect. Illustratively, the computing device cluster can comprise one or more computing devices.

[0030] In a fifth aspect, the present application provides a computer program product comprising instructions, when the instructions are executed by a computing device cluster, the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect. Illustratively, the computing device cluster can comprise one or more computing devices.

[0031] It can be understood that the beneficial effects of the above-mentioned second aspect to fifth aspect can refer to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0032] FIG. 1 is a two-fork form of a backbone network of a task-level dynamic network under a GPPF framework according to an embodiment of the present application;

[0033] FIG. 2 is a structural schematic diagram of a multi-task network according to an embodiment of the present application;

[0034] FIG. 3 is a structural schematic diagram of a backbone network according to an embodiment of the present application;

[0035] FIG. 4 is a schematic diagram of a backbone network under different inference strategies according to an embodiment of the present application;

[0036] FIG. 5 is a flowchart of a multi-task network training method according to an embodiment of the present application;

[0037] FIG. 6 is a process diagram for adjusting values of elements in a one-hot mask matrix based on a gradient of a backhaul-to-network layer according to an embodiment of the present application;

[0038] FIG. 7 is a process diagram for calculating a new gradient of a network layer with a weight matrix according to an embodiment of the present application;

[0039] FIG. 8 is a flow diagram of a task inference method according to an embodiment of the present application;

[0040] FIG. 9 is a structural diagram of a multi-task network training apparatus according to an embodiment of the present application;

[0041] FIG. 10 is a structural diagram of a computing device according to an embodiment of the present application;

[0042] FIG. 11 is a structural diagram of a computing device cluster according to an embodiment of the present application;

[0043] FIG. 12 is a structural diagram of another computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The term "and / or" used in this document is used to describe an association relationship between associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol " / " in this document represents an or relationship between associated objects, for example, A / B represents A or B.

[0045] The terms "first" and "second" and the like in the description and claims of this document are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, and are not used to describe a specific order of the response messages.

[0046] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present relevant concepts in a concrete manner.

[0047] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0048] Generally, in the field of multi-task learning, task conflicts can be alleviated in multi-task learning by architecture-based multi-task learning methods to improve the accuracy of neural network models on multiple tasks. Among them, the architecture-based multi-task learning method mainly tries to design or learn a network that can balance between shared parameters and task-specific parameters to alleviate task conflicts. The main difficulties faced by such methods are how to design the sharing strategy of parameters, how to determine the proportion of shared parameters, and how to parameterize and combine shared and task-specific modules. In such learning methods, a task-level dynamic network under a general perception pre-training framework (GPPF) can be used to solve task conflicts and domain conflicts, and to achieve effective knowledge sharing. During inference, the total parameter amount can be greatly reduced by reusing modules between tasks. Currently, as shown in FIG. 1, the backbone of the task-level dynamic network under the GPPF framework can generally have two fork forms, which are the high-level fork form shown in (A) of FIG. 1 and the bottom-level fork form shown in (B) of FIG. 1. In the high fork form, during the inference process, the shared part is used to infer data in the low layer, and then the inference result of the shared part is transmitted to the modules associated with different tasks in the high layer for inference. Although this method can solve the task conflict problem, most of the inference uses the shared part, which limits the inference accuracy. In the low fork form, during the inference process, different modules associated with different tasks are used to infer data in the low layer, and then the shared part is used to infer the inference results of the modules associated with different tasks in the high layer. Although this method can solve the task conflict problem, it needs to use the shared part to infer each task separately, which cannot be parallel inference, thereby increasing the inference computing power.

[0049] The above multi-task learning method alleviates task conflicts from the module level, which can alleviate task conflicts, but has limited ability to solve task conflicts and has many disadvantages. In view of this, the embodiment of the present application provides a multi-task learning method, which constructs the mapping relationship between tasks and channels (i.e., "output channels"), so that different tasks can use different channels of network layers during training and inference, thereby solving task conflicts from the channel level in multi-task learning. In addition, this multi-task learning method can also store the features of different tasks in different channels of the feature map, so that parallel inference of different tasks can be performed at the channel level to achieve the purpose of ensuring inference accuracy and reducing inference computing power. The technical solutions provided by the embodiments of the present application are introduced below.

[0050] Exemplarily, FIG. 2 shows a structural schematic diagram of a multi-task network provided by an embodiment of the present application. As shown in FIG. 2, the multi-task network 200 can include a backbone 210 and a plurality of head networks 220. The backbone 210 is a common part of the multi-task network 200 for extracting features, and is mainly responsible for extracting features from input data, which can be used for different tasks, such as image classification, object detection, and semantic segmentation, etc. Exemplarily, as shown in FIG. 3, the backbone 210 can include a plurality of network layers (such as convolutional layers, fully connected layers, pooling layers, activation layers, etc.), and each network layer can include a plurality of neurons. The neurons between network layers can be fully connected or partially connected. For the connected neurons, the output of the neurons of the previous layer can be used as the input of the neurons of the next layer or other layers. In the embodiment, one or more network layers in the backbone 210 can have a weight matrix, such as one network layer having a weight matrix, two network layers having a weight matrix, etc. The weight matrix of a network layer defines the connection weights between the neurons of the layer. Assuming that there is a fully connected layer including n input neurons and m output neurons, the shape of the weight matrix is (m, n), i.e., m rows and n columns. Exemplarily, the network layer with the weight matrix can be a fully connected layer, a convolutional layer, etc. In addition, in the backbone 210, the network layer with the weight matrix can have a task-related one-hot mask matrix wherein T is the number of tasks, C in is the number of input channels of the network layer, and C out is the number of output channels of the network layer. It should be understood that the one-hot mask matrix is a form of one-hot encoding, which is a multi-dimensional array or matrix composed of 0 or 1. The one-hot mask matrix M conv includes two or more sub-matrices wherein i ∈ [1, T], when i = 1, it indicates a sub-matrix related to the first task; when i = 5, it indicates a sub-matrix related to the fifth task. The one-hot mask matrix M conv indicates the situation of the parameters in the network layer used by each task. The elements in the one-hot mask matrix M conv have a value of 0 or 1. Through the one-hot mask matrix M conv , a part of the parameters in the network layer can be masked. When the value of an element in the one-hot mask matrix M conv is 0, it indicates that the parameters in the network layer related to the element are masked (i.e., not used); when the value of an element in the one-hot mask matrix M convWhen the value of an element in the one-hot mask matrix M conv may be based on and . Wherein, is the mask matrix of the output channels of the network layer, M Cin is the mask matrix of the input channels of the network layer. The mask matrix M of a network layer can inherit the mask matrix M of the previous network layer. conv The output of the network layer with the one-hot mask matrix M wherein, y is the output of the network layer, W is the weight matrix of the network layer, x is the input of the network layer, B is the bias term, W and M conv need to be bit multiplied, and B and M Cout need to be bit multiplied. Exemplarily, the bit multiplication can also be referred to as element-wise multiplication, which means that the elements at the same position in different matrices are multiplied. For example, if the matrix the matrix and the matrix If the matrix A in the example is understood as a sub-matrix in the one-hot mask matrix, and the matrix B is understood as the parameters in the network layer, then some parameters in the matrix B can be filtered out (i.e., masked out) by bit multiplication, so that the subsequent inference can not use the masked parameters. Thus, by using the one-hot mask matrix M conv , the required parameters for each task can be selected from the parameters of the network layer during inference, so that different tasks can be inferred by different output channels in the network layer. Exemplarily, the backbone network 220 can be but is not limited to a convolutional neural network (CNN). In this embodiment, the inference strategy of the backbone network 220 can include a read-only multi-task inference strategy and a partial order multi-task inference strategy. Under the read-only multi-task inference strategy, each task can use the channels of other tasks for inference during feedforward, that is, the channel mask (which can also be referred to as a sub-matrix related to the i-th task) used by the i-th task during feedforward is the logical OR of all task channel masks (which can also be referred to as sub-matrices related to each task), Exemplarily, the logical OR of two sub-matrices in the one-hot mask matrix can be understood as follows: when the elements at the same position in the two sub-matrices have one value of “1”, the element at the position is taken as “1”; if the values of the elements at the same position in the two sub-matrices are both “0”, the element at the position is taken as “0”. For example, if the sub-matrix the sub-matrix The logical OR of the two is Wherein, using the logical OR of each sub-matrix in the one-hot mask matrix to do the feedforward for each task can be understood as using the logical OR of each sub-matrix in the one-hot mask matrix as a new sub-matrix associated with each task. Then, the new sub-matrix can be used to process each task. Each task can only update the weight of its own channel during gradient backpropagation, i.e., the i-th task updates the weight of its own channel according to its own mask after gradient backpropagation. Update the parameters. For example, as shown in (A) of FIG. 4, taking a certain one network layer 41 of the backbone network 220 as an example, during the feedforward, task 1 and task 2 use the logical OR of the channels of the two to reason; during the feedback, each task only updates the weight of its own channel. Under the partial order multi-task reasoning strategy, each network layer with a weight matrix in the backbone network 220 can include: a shared channel. In addition, the i-th non-shared channel in the k-th network layer with a weight matrix is connected with the shared channel in the (k-1)-th network layer and other channels containing the i-th non-shared channel. Wherein, the non-shared channels in the (k-1)-th network layer containing the i-th non-shared channel in the k-th network layer means that the task set using the non-shared channels in the (k-1)-th network layer contains the task set using the i-th non-shared channel in the k-th network layer. For example, as shown in (B) of FIG. 4, the k-th network layer includes: a shared channel, a shared channel of task 1 and task 2, a private channel of task 1, a private channel of task 2, and a private channel of task 3; the (k-1)-th network layer includes: a shared channel, a shared channel of task 1 and task 2, and a private channel of task 1. At this time, the shared channel in the (k-1)-th network layer can be connected with each channel in the k-th network layer, the shared channel of task 1 and task 2 in the (k-1)-th network layer can be connected with the shared channel of task 1 and task 2, the private channel of task 1, and the private channel of task 2 in the k-th network layer, and the private channel of task 1 in the (k-1)-th network layer can be connected with the private channel of task 1 in the k-th network layer. As can be seen from (B) of FIG. 4, under the partial order multi-task reasoning strategy, the task knowledge can only flow from the high-order channel to the low-order channel during the feedforward of the backbone network 120. Wherein, the higher-order knowledge sharing degree of channel 1 than channel 2 means that the task set using channel 1 contains the task set using channel 2. In addition, under the partial order multi-task reasoning strategy, the newly added task can use the learned shared knowledge, and only needs to add a small amount of parameters to learn the exclusive knowledge of the new task. For example: continuing to refer to (B) of FIG. 4, assuming that there is no channel of task 3 in the k-th network layer, when task 3 needs to be added, only a small amount of parameters need to be added to the k-th network layer to learn the knowledge of task 3.

[0051] The head network 220 is a dedicated part of the multi-task network 200 for a specific task, and is mainly responsible for performing a specific task according to the features extracted by the backbone network 210, such as object bounding box prediction in target detection, pixel classification in semantic segmentation, etc. The structure of the head network 220 is related to the task and needs to be designed according to different tasks. For example, the head network 220 can be multiple. In addition, the head network 220 can be, but is not limited to, a multilayer perceptron (MLP), a CNN, etc.

[0052] In the multi-task network 200 shown in FIG. 1, since the sub-matrix in the one-hot mask matrix can be used to mask at least part of the parameters in the network layer, after the adjustment of the element value in the one-hot mask matrix is completed, different sub-matrices can mask different parameters in the network layer. Each sub-matrix in the one-hot mask matrix is related to a task, so that different parameters in the network layer can be used to train or infer different tasks when different tasks are inferred, which establishes the mapping relationship between the task and the channel. Further, since the one-hot mask matrix indicating the mapping relationship between the task and the channel is allocated in the backbone network 210, multi-task learning at the channel level can be realized. In this way, in the multi-task scenario, different tasks use different sub-networks in the backbone network (i.e., each network layer uses different channels (i.e., “output channels”) for inference, which helps to alleviate the conflict in multi-task learning. In addition, while maintaining the use of different sub-networks by different tasks, the features of different tasks are spliced at the channel level to realize parallel inference at the channel level, so that the inference accuracy can be guaranteed and the inference computing power can be reduced. In addition, since multiple tasks can be learned in a network layer in the backbone network, and the parameters in the same network layer can be reused by multiple tasks, the relationship changes between different tasks can be conveniently captured in a network layer, so that the conditions of overfitting, underfitting or failure to learn enough knowledge in the training process can be reduced, and the inference accuracy can be improved.

[0053] The training process of the multi-task network 200 shown in FIG. 2 is described below.

[0054] Exemplarily, FIG. 5 shows a flow diagram of a multi-task network training method provided by the embodiments of the present application. It can be understood that the method can be executed by any device, equipment, platform, or cluster of equipment with computing and processing capabilities. The multi-task network can include a backbone network. Of course, the multi-task network can also include head networks, and different head networks are specific parts of different tasks, while the backbone network is a common part of different tasks. The backbone network includes network layers with weight matrices, and the network layers have one-hot mask matrices. Exemplarily, each network layer with a weight matrix can have one one-hot mask matrix, or a part of network layers with weight matrices can have one one-hot mask matrix respectively, which is not limited here. The one-hot mask matrix can include T (T≥2) sub-matrices, and one sub-matrix can be related to one task, and the sub-matrix can be used to mask the parameters in the network layer. As shown in FIG. 5, the multi-task network training method can include the following steps.

[0055] S501, obtaining the gradients of T (T≥2) tasks back to the network layer with the weight matrix.

[0056] In this embodiment, when training the multi-task network, the training data can be input into the backbone network, and the training data can be processed by the backbone network. During the processing of the backbone network, the network layer with the one-hot mask matrix can use the one-hot mask matrix to perform bit multiplication with the parameters in the network layer. For example, the one-hot mask matrix can be used to perform bit multiplication with the weight matrix of the network layer, and the mask matrix of the output channel contained in the one-hot mask matrix can be used to perform bit multiplication with the bias term of the network layer, etc. For the bit multiplication, refer to the description above, which will not be repeated here. After the processing of the backbone network is completed, the processing result of the backbone network can be transmitted to different head networks for subsequent processing by the head networks. Then, the loss corresponding to each task can be obtained by calculating the processing result of each head network and the corresponding label by using the loss function. Finally, the gradients of each task back to the network layer with the weight matrix can be obtained by calculating the determined loss by using the gradient back propagation algorithm, etc. In some embodiments, when the read-only inference strategy is used, the backbone network can use the logical OR of each sub-matrix in the one-hot mask matrix associated with the network layer with the weight matrix to process the task in the T tasks during the processing. That is, the logical OR of each sub-matrix in the one-hot mask matrix can be used as a new sub-matrix associated with each task in the T tasks, and then the new sub-matrix is used to process each task in the T tasks.

[0057] S502, adjusting the value of the element in the one-hot mask matrix associated with the network layer with the weight matrix based on the gradients back to the network layer with the weight matrix, wherein the initial value of the element in the one-hot mask matrix is 1.

[0058] In this embodiment, after obtaining the gradient returned to the network layer with the weight matrix, the value of the element in the one-hot mask matrix associated with the network layer can be adjusted through the gradient. Specifically, as shown in FIG. 6, the following steps can be included: in S601, based on the gradient returned by T tasks to each channel (i.e., “output channel”) in the network layer with the weight matrix, the weight of each channel (i.e., “output channel”) in the network layer assigned to each task of the T tasks is determined respectively. In this step, two factors are mainly considered: 1) the importance of the channel to different tasks, and 2) the conflict degree of different tasks on the channel. For the first factor, for any one channel (hereinafter referred to as “the cth channel”), the disturbance loss degree of the cth channel to the ith task can be calculated based on the gradient returned by the cth channel to the ith task and the parameters of the cth channel. Exemplarily, but not limited to, the calculation formula can be: disturbance loss degree Or, And so on. Wherein, is the gradient returned by the ith task to the cth channel, w i c is the parameter of the cth channel. Wherein, the greater the calculated disturbance loss degree, the more important the cth channel is to the optimization of the loss of the ith task. For the second factor, the gradient similarity of the ith task and other tasks on the cth channel can be calculated based on the gradient returned by the ith task to the cth channel and the gradient returned by other tasks except the ith task to the cth channel. Exemplarily, the gradient similarity of the two tasks on the cth channel can be calculated by calculating the gradient of the ith task and the gradient of any other task through a similarity algorithm such as cosine similarity. Wherein, the greater the gradient similarity, the smaller the conflict of the two tasks on the cth channel, and the more likely the cth channel is assigned to the ith task. After calculating the disturbance loss degree of the cth channel to the ith task and the gradient similarity of the ith task and other tasks on the cth channel, the weight of the cth channel assigned to the ith task can be calculated through the calculated results. Exemplarily, but not limited to, the calculation formula can be: weight of the cth channel assigned to the ith task Wherein, is the gradient similarity of the ith task and the jth task on the cth channel, and a is a hyperparameter.

[0059] At S602, values of elements in the one-hot mask matrix associated with the network layer are adjusted based on the weights of the respective channels in the network layer assigned to the respective tasks of the T tasks. After the weights of the respective channels assigned to the respective tasks are calculated, for the c-th channel, quantiles of the weights of the c-th channel assigned to the respective tasks can be calculated. Of course, the quantiles can be replaced by other thresholds, which are not limited here. Then, values of elements in the respective sub-matrices associated with the c-th channel in the one-hot mask matrix are adjusted by the quantiles. Wherein, when the weight of the c-th channel assigned to the i-th task is greater than or equal to the calculated quantile, the value of the element associated with the c-th channel in the i-th sub-matrix (i.e., the sub-matrix associated with the i-th task) can be set to 1, so that the c-th channel is assigned to the i-th task. When the weight of the c-th channel assigned to the i-th task is less than the calculated quantile, the value of the element associated with the c-th channel in the i-th sub-matrix (i.e., the sub-matrix associated with the i-th task) can be set to 0, so that the c-th channel is not assigned to the i-th task. Exemplarily, the initial values of the elements in the one-hot mask matrix can all be 1, or some can be 1 and some can be 0, or all can be 0, which are not limited here. Wherein, setting the value of the element associated with the c-th channel in the i-th sub-matrix to 1 can be understood as adjusting the value of the element from 0 to 1, or keeping the value of the element as 1. Setting the value of the element associated with the c-th channel in the i-th sub-matrix to 0 can be understood as adjusting the value of the element from 1 to 0, or keeping the value of the element as 0.

[0060] After the adjustment of the values of the elements in the one-hot mask matrix is completed, S503 can be executed.

[0061] S503, based on the adjusted one-hot mask matrix associated with the network layer having the weight matrix and the gradient backpropagated to the network layer having the weight matrix, updating the parameters of the backbone network.

[0062] In this embodiment, after the adjustment of the values of the elements in the one-hot mask matrix is completed, the new gradient of the network layer having the weight matrix can be calculated based on the adjusted one-hot mask matrix associated with the network layer having the weight matrix and the gradient backpropagated to the network layer having the weight matrix. Then, the parameters in the backbone network are updated based on the calculated new gradient of the network layer having the weight matrix, such as updating the parameters in the backbone network in the direction of gradient descent to train the backbone network.

[0063] As a possible implementation, as shown in FIG. 7, the process of calculating the new gradient of the network layer with the weight matrix can include the following steps: at S701, grouping the channels in the network layer with the weight matrix based on the adjusted one-hot mask matrix associated with the network layer with the weight matrix to obtain N (N≥2) channel groups, wherein different channel groups process different sets of tasks. Specifically, the tasks processed by different channels in each network layer can be determined by the one-hot mask matrix associated with each network layer. Then, the grouping of the channels in the network layer can be completed by dividing the channels processing the same set of tasks into a group. For example, if the cth channel in network layer 1 is assigned to task 1 and task 2, the dth channel is assigned to task 2, task 3 and task 4, the e th channel in network layer 2 is assigned to task 1, the f th channel is assigned to task 1 and task 2, and the g th channel is assigned to task 2, task 3 and task 4, the c th channel and the f th channel can be divided into a group, the d th channel and the g th channel can be divided into a group, and the e th channel can be divided into a group by itself. At S702, the gradient modulation algorithm, such as the Nash-MTL method, the Aligned-MTL method, etc., is used to calculate the gradient backpropagated to the channels included in each channel group to obtain the weighted weight of each channel group wherein α i is the weighted weight of the i th channel group. At S703, based on the weighted weight of each channel group and the gradient backpropagated to the network layer with the weight matrix, the new gradient of the network layer with the weight matrix is calculated. For example, the formula for calculating the new gradient of the network layer with the weight matrix can be, but is not limited to, new gradient wherein G i is the gradient of the i th channel group collected, which can be calculated by adding the gradient on the channels included in the i th channel group, etc.

[0064] After the new gradient is calculated, the parameters in the backbone network can be updated in the direction of the new gradient descending to complete the training of the multi-task network. Since the update of the one-hot mask matrix associated with the network layer with the weight matrix is completed during the training process, and the one-hot mask matrix can indicate the parameters in the network layer used by each task, the allocation of the mapping relationship between the tasks and the channels in the multi-task network can be completed after the training is completed.

[0065] It should be understood that since the head network in the multi-task network is for a specific task, the training of the head network can update the parameters in the head network based on the gradient backpropagated to the head network corresponding to the specific task to complete the training of the head network.

[0066] After the training of the multi-task network is completed, each obtained one-hot mask matrix can be used to do bit multiplication with the parameters of the corresponding network layer to update the parameters (such as weights and / or bias terms, etc.) in the network layer. In this way, the one-hot mask matrix can be discarded directly in the subsequent inference, and the multi-task learning network can be directly used for inference to avoid introducing additional overheads caused by the one-hot mask matrix. Of course, when inference is needed subsequently, each obtained one-hot mask matrix can be used to do bit multiplication with the parameters of the corresponding network layer to update the parameters in the network layer during loading of the multi-task network, so that the one-hot mask matrix can be discarded directly in the subsequent inference, and the multi-task learning network can be directly used for inference to avoid introducing additional overheads caused by the one-hot mask matrix.

[0067] The process of using the foregoing multi-task learning network for inference will be introduced below.

[0068] For example, FIG. 8 shows a flowchart of a task inference method provided by an embodiment of the present application. It can be understood that the method can be executed by any device, equipment, platform, or device cluster having computing and processing capabilities. As shown in FIG. 8, the task inference method can include the following steps:

[0069] S801, obtaining user data.

[0070] In this embodiment, the user data can be data uploaded by a user or data uploaded by equipment, which is not limited here. For example, the user data can be an image or a video, etc.

[0071] S802, using a multi-task network to infer the user data; wherein during loading of the multi-task network, each one-hot mask matrix is used to do bit multiplication with the parameters of the corresponding network layer, and the results obtained by the bit multiplication are used to perform parallel inference on each task in the current inference during the inference.

[0072] In this embodiment, the user data can be input to the foregoing multi-task network whose training is completed, so as to use the multi-task network to infer the user data. During loading of the multi-task network, each one-hot mask matrix can be used to do bit multiplication with the parameters of the corresponding network layer, and the results obtained by the bit multiplication can be used to perform parallel inference on each task during the inference. For example, after the one-hot mask matrix is used for bit multiplication, the mask matrix can be discarded. For example, the number of tasks in the current inference can be less than or equal to the number of the foregoing T tasks. For example, when the number of tasks used during training is 10, the number of tasks in each inference during the inference process is less than or equal to 10.

[0073] S803, outputting the inference results of each task in the current inference.

[0074] In this embodiment, the multi-task network can output the inference results of each task in this inference after completing the inference.

[0075] In this way, the task inference is completed.

[0076] It can be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the above-described various embodiments can be combined according to actual conditions, and the combined scheme is still within the protection scope of the present application.

[0077] Based on the method in the above embodiment, the embodiments of the present application further provide a multi-task network training device.

[0078] For example, FIG. 9 shows a structural schematic diagram of a multi-task network training device provided by the embodiments of the present application. The multi-task network includes a backbone network, and the backbone network includes a network layer with a weight matrix, the network layer has a one-hot mask matrix, the one-hot mask matrix includes T (T≥2) sub-matrices, each of the T sub-matrices is related to a task, and the sub-matrix is used to mask the parameters in the network layer. As shown in FIG. 9, the multi-task network training device 900 can include an acquisition module 901 and a processing module 902. The acquisition module 901 is configured to acquire the gradients of the T tasks returned to the network layer. The processing module 902 is configured to adjust the values of the elements in the one-hot mask matrix associated with the network layer based on the gradients returned to the network layer, and update the parameters of the backbone network based on the adjusted one-hot mask matrix and the gradients returned to the network layer. For example, the initial value of the element in the one-hot mask matrix is 1.

[0079] In some embodiments, when the processing module 902 adjusts the values of the elements in the one-hot mask matrix associated with the network layer based on the gradients returned to the network layer, it is specifically configured to determine the weights of each channel in the network layer allocated to each task in the T tasks based on the gradients of each channel in the network layer returned to the network layer by each task in the T tasks, respectively; and adjust the values of the elements in the one-hot mask matrix associated with the network layer based on the weights of each channel in the network layer allocated to each task in the T tasks.

[0080] In some embodiments, the processing module 902, when determining the weight of each channel in the network layer assigned to each of the T tasks based on the gradients of the T tasks backpropagated to each channel in the network layer, respectively, is specifically configured to: calculate a loss disturbance degree of any one channel in the network layer to a first task based on the parameters of the any one channel and the gradient of the first task backpropagated to the any one channel, the first task being any one of the T tasks; calculate a gradient similarity of the first task and other tasks on the any one channel based on the gradient of the first task backpropagated to the any one channel and the gradients of the other tasks than the first task backpropagated to the any one channel; and calculate the weight of the any one channel assigned to the first task based on the loss disturbance degree and the gradient similarity.

[0081] In some embodiments, the processing module 902, when adjusting the values of the elements in the one-hot mask matrix associated with the network layer based on the weights of each channel in the network layer assigned to each of the T tasks, is specifically configured to: set the value of the element in the first sub-matrix related to the any one channel to 0 in the case that the weight of the any one channel in the network layer assigned to the first task is less than a first threshold, the first task being any one of the T tasks, and the first sub-matrix being the matrix related to the first task in the one-hot mask matrix associated with the network layer; and set the value of the element in the first sub-matrix related to the any one channel to 1 in the case that the weight of the any one channel in the network layer assigned to the first task is greater than or equal to the first threshold.

[0082] In some embodiments, the processing module 902, when updating the parameters of the backbone network based on the adjusted one-hot mask matrix associated with the network layer and the gradients backpropagated to the network layer, is specifically configured to: calculate new gradients of the network layer based on the adjusted one-hot mask matrix associated with the network layer and the gradients backpropagated to the network layer; and update the parameters of the backbone network based on the new gradients of the network layer.

[0083] In some embodiments, the processing module 902, when calculating new gradients of the network layer based on the adjusted one-hot mask matrix associated with the network layer and the gradients backpropagated to the network layer, is specifically configured to: group the channels in the network layer based on the adjusted one-hot mask matrix associated with the network layer to obtain N (N≥2) channel groups, wherein the task sets processed by different channel groups are different; calculate the weighted weights of each channel group by respectively calculating the gradients on the channels included in each channel group using a gradient modulation algorithm; and calculate the new gradients of the network layer based on the weighted weights of each channel group and the gradients backpropagated to the network layer.

[0084] In some embodiments, the one-hot mask matrix is used for bit multiplication with the parameters of the corresponding network layer.

[0085] In some embodiments, before the obtaining module 901 obtains the gradients of the T tasks returned to the network layer of the backbone network, the processing module 902 is further configured to: take the logical OR of each sub-matrix in the one-hot mask matrix associated with the network layer as a new sub-matrix associated with each of the T tasks, and perform inference on each of the T tasks based on the new sub-matrix.

[0086] In some embodiments, there are multiple network layers with weight matrices, each of which includes shared channels, and at least one of which further includes non-shared channels; wherein the ith non-shared channel in the kth network layer with a weight matrix is connected to the shared channel in the (k-1)th network layer with a weight matrix and other channels containing the ith non-shared channel.

[0087] In some embodiments, after the training of the multi-task network is completed, the obtaining module 901 is further configured to obtain user data. The processing module 902 is further configured to perform inference on the user data using the multi-task network; wherein during loading of the multi-task network, each one-hot mask matrix is multiplied with the parameters of the corresponding network layer, and the results of the bit multiplication are used to perform parallel inference on each task in the current inference during the inference; and the inference results of each task in the current inference are output.

[0088] In some embodiments, the obtaining module 901 and the processing module 902 shown in FIG. 9 can be implemented by software or by hardware. For example, the implementation of the obtaining module 901 is described below. Similarly, the implementation of the processing module 902 can refer to the implementation of the obtaining module 901.

[0089] As an example of a software functional unit, the obtaining module 901 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the obtaining module 901 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region), or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes a data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0090] Similarly, the plurality of hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually one VPC is set in one region, and communication between two VPCs in the same region and between VPCs in different regions needs to set a communication gateway in each VPC to realize the interconnection between VPCs through the communication gateway.

[0091] As an example of a hardware functional unit, the obtaining module 901 can include at least one computing device, such as a server, etc. Alternatively, the obtaining module 901 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. Among them, the above-mentioned PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0092] The plurality of computing devices included in the obtaining module 901 can be distributed in the same region or in different regions. The plurality of computing devices included in the obtaining module 901 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the obtaining module 901 can be distributed in the same VPC or in multiple VPCs. Among them, the plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0093] It should be noted that in other embodiments, the obtaining module 901 can be used to perform any step of the multi-task network training method described in the above embodiments, and the processing module 902 can also be used to perform any step of the multi-task network training method described in the above embodiments. In addition, the obtaining module 901 can also be combined with the processing module 902 to be responsible for performing any step of the multi-task network training method described in the above embodiments. In addition, the steps implemented by the obtaining module 901 and the processing module 902 can also be specified as needed, and the obtaining module 901 and the processing module 902 respectively implement different steps of the multi-task network training method described in the above embodiments to realize the entire function of the multi-task network training device 500 shown in FIG. 9.

[0094] The present application also provides a computing device 1000. As shown in FIG. 10, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other through the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1000.

[0095] The bus 1002 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 10, but it does not mean that there is only one bus or only one type of bus. The bus 1004 can include a path for transmitting information between various components (e.g., the memory 1006, the processor 1004, the communication interface 1008) of the computing device 1000.

[0096] The processor 1004 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0097] The memory 1006 can include a volatile memory (e.g., a random access memory (RAM)) and a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0098] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to respectively implement the functions of the acquisition module 901 and the processing module 902 shown in FIG. 9, thereby implementing the multi-task network training method described in the above embodiments. That is, the memory 1006 stores instructions for executing the multi-task network training method described in the above embodiments.

[0099] Alternatively, the memory 1006 stores executable code that, when executed by the processor 1004, implements the functionality of the multi-task network training apparatus 500 shown in FIG. 9, respectively, to implement the multi-task network training method described in the above embodiments. That is, the memory 1006 stores instructions for implementing the multi-task network training method described in the above embodiments.

[0100] The communication interface 1008 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, and the like to enable communication between the computing device 1000 and other devices or communication networks.

[0101] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0102] As shown in FIG. 11, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster can store the same instructions for implementing the multi-task network training method described in the above embodiments.

[0103] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster can also respectively store partial instructions for implementing the multi-task network training method described in the above embodiments. In other words, the combination of one or more computing devices 1000 can collectively execute the instructions for implementing the multi-task network training method described in the above embodiments.

[0104] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions for respectively implementing part of the functionality of the multi-task network training apparatus 900 shown in FIG. 9. That is, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functionality of one or more of the obtaining module 901 and the processing module 902.

[0105] In some possible implementation manners, one or more of the computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, or the like. FIG. 12 shows a possible implementation manner. As shown in FIG. 12, two computing devices 1000A and 1000B are connected through a network. Specifically, the computing devices are connected to the network through communication interfaces in the computing devices. In this kind of possible implementation manner, the memory 1006 in the computing device 1000A stores instructions for performing the functions of the obtaining module 901. Meanwhile, the memory 1006 in the computing device 1000B stores instructions for performing the functions of the processing module 902.

[0106] It should be understood that the functions of the computing device 1000A shown in FIG. 12 can also be completed by multiple computing devices 1000. Similarly, the functions of the computing device 1000B can also be completed by multiple computing devices 1000.

[0107] The embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manners of the computing device clusters described with reference to FIG. 11 and FIG. 12. The difference is that the memory 1006 in one or more of the computing devices 1000 in the computing device cluster can store the same instructions for performing the method in the foregoing embodiments.

[0108] In some possible implementation manners, the memory 1006 of one or more of the computing devices 1000 in the computing device cluster can also respectively store partial instructions for performing the multi-task network training method. In other words, the combination of one or more of the computing devices 1000 can collectively execute the instructions for performing the multi-task network training method.

[0109] Based on the method in the foregoing embodiments, the embodiments of the present application provide a computer-readable storage medium including computer program instructions, when the computer program instructions are executed by a computing device cluster including at least one computing device, the computing device cluster executes the method in the foregoing embodiments. For example, the computer-readable storage medium can be any available medium or a data storage device including one or more available media that a computing device is capable of storing. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), or the like.

[0110] Based on the method in the foregoing embodiments, the embodiments of the present application provide a computer program product including instructions, when the instructions are run by a computing device, the method in the foregoing embodiments is executed by a computing device cluster including at least one computing device.

[0111] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0112] The method steps in the embodiments of the present application can be realized by hardware or by the processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0113] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0114] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application.

[0115] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-task network training method, characterized in that: The multi-task network comprises: a backbone network comprising: a network layer having a weight matrix, the network layer having a one-hot mask matrix, the one-hot mask matrix comprising: T sub-matrices, wherein T≥2, each of the T sub-matrices being associated with one task, the sub-matrices being used to mask parameters in the network layer; The method comprises: obtaining gradients of T tasks back to the network layer; based on the gradients back to the network layer, adjusting the values of elements in the one-hot mask matrix associated with the network layer; based on the adjusted one-hot mask matrix and the gradients back to the network layer, updating the parameters of the backbone network.

2. The method of claim 1, wherein, The method comprises: based on the gradients of the T tasks back to each channel in the network layer, respectively determining the weights of each channel in the network layer assigned to each of the T tasks; based on the weights of each channel in the network layer assigned to each of the T tasks, adjusting the values of elements in the one-hot mask matrix associated with the network layer.

3. The method of claim 2, wherein, The method comprises: based on the parameters of any one channel in the network layer and the gradient of a first task back to the any one channel, calculating the loss disturbance degree of the any one channel to the first task, the first task being any one of the T tasks; based on the gradient of the first task back to the any one channel and the gradients of other tasks except the first task among the T tasks back to the any one channel, calculating the gradient similarity of the first task and the other tasks on the any one channel; based on the loss disturbance degree and the gradient similarity, calculating the weight of the any one channel assigned to the first task.

4. The method according to claim 2 or 3, characterized in that, The method comprises: in the case that the weight of any one channel in the network layer assigned to a first task is less than a first threshold, setting the value of an element in a first sub-matrix associated with the any one channel to 0, the first task being any one of the T tasks, the first sub-matrix being a matrix associated with the first task in the one-hot mask matrix associated with the network layer; in the case that the weight of any one channel in the network layer assigned to the first task is greater than or equal to the first threshold, setting the value of the element in the first sub-matrix associated with the any one channel to 1.

5. The method according to any of claims 1 to 4, characterized in that The method comprises: calculating a new gradient of the network layer based on the adjusted one-hot mask matrix associated with the network layer and the gradient backpropagated to the network layer; updating parameters of the backbone network based on the new gradient of the network layer.

6. The method of claim 5, wherein, The calculating a new gradient of the network layer based on the adjusted one-hot mask matrix associated with the network layer and the gradient backpropagated to the network layer comprises: grouping channels in the network layer based on the adjusted one-hot mask matrix associated with the network layer to obtain N channel groups, where N≥2, and different channel groups process different sets of tasks; using a gradient modulation algorithm to calculate a gradient on each channel included in each of the N channel groups to obtain a weighted weight of each channel group; calculating a new gradient of the network layer based on the weighted weight of each channel group and the gradient backpropagated to the network layer.

7. The method according to any of claims 1 to 6, characterized in that The one-hot mask matrix is used for bit multiplication with parameters of the network layer.

8. The method according to any one of claims 1 to 7, characterized in that, Before the gradient of the T tasks backpropagated to the network layer in the backbone network is obtained, the method further comprises: performing processing on each of the T tasks based on a new sub-matrix associated with each task in the T tasks, where the new sub-matrix is obtained by performing a logical OR operation on each sub-matrix in the one-hot mask matrix.

9. The method according to any of claims 1 to 7, characterized in that There are a plurality of network layers with weight matrices, and each network layer with a weight matrix comprises shared channels and at least one network layer with a weight matrix further comprises non-shared channels. In the kth network layer with a weight matrix, the ith non-shared channel is connected to a shared channel in the (k-1)th network layer with a weight matrix and other channels including the ith non-shared channel.

10. The method according to any one of claims 1 to 9, characterized in that, An initial value of an element in the one-hot mask matrix is 1.

11. The method according to any one of claims 1 to 10, characterized in that, After the training of the multi-task network is completed, the method further comprises: obtaining user data; performing inference on the user data by using the multi-task network; wherein during loading of the multi-task network, each one-hot mask matrix is used for bit multiplication with parameters of a corresponding network layer, and during inference, a result obtained by bit multiplication is used for parallel inference on each task in this inference; outputting inference results of each task in this inference.

12. A multi-task network training apparatus, comprising: The multi-task network comprises a backbone network, and the backbone network comprises network layers with weight matrices, each network layer with a weight matrix having a one-hot mask matrix, the one-hot mask matrix comprising T sub-matrices, where T≥2, each sub-matrix in the T sub-matrices being associated with a task, and the sub-matrix being used for masking parameters in the network layer. The apparatus comprises: a obtaining module configured to obtain a gradient of T tasks backpropagated to the network layer; a processing module configured to adjust a value of an element in a one-hot mask matrix associated with the network layer based on the gradient backpropagated to the network layer; the processing module is further configured to update parameters of the backbone network based on the adjusted one-hot mask matrix and the gradient backpropagated to the network layer.

13. The apparatus of claim 12, wherein, The processing module is specifically configured to: based on the gradients of the T tasks returned to each channel in the network layer, respectively determine the weights of each channel in the network layer allocated to each of the T tasks; based on the weights of each channel in the network layer allocated to each of the T tasks, adjust the values of elements in the one-hot mask matrix associated with the network layer.

14. The apparatus of claim 13, wherein, The processing module is specifically configured to: based on the parameters of any one channel in the network layer and the gradient of the first task returned to the any one channel, calculate the loss disturbance degree of the first task removed from the any one channel, the first task being any one of the T tasks; based on the gradient of the first task returned to the any one channel and the gradient of other tasks in the T tasks returned to the any one channel, calculate the gradient similarity of the first task and the other tasks on the any one channel; based on the loss disturbance degree and the gradient similarity, calculate the weight of the any one channel allocated to the first task.

15. The apparatus of claim 13 or 14, wherein, The processing module is specifically configured to: in a case where the weight of any one channel in the network layer allocated to the first task is less than a first threshold, set the value of the element related to the any one channel in the first sub-matrix to 0, the first task being any one of the T tasks, and the first sub-matrix being a matrix related to the first task in the one-hot mask matrix associated with the network layer; in a case where the weight of any one channel in the network layer allocated to the first task is greater than or equal to the first threshold, set the value of the element related to the any one channel in the first sub-matrix to 1.

16. The apparatus of any of claims 12-15, wherein, The processing module is specifically configured to: based on the adjusted one-hot mask matrix associated with the network layer and the gradient returned to the network layer, calculate new gradients of the network layer; based on the new gradients of the network layer, update the parameters of the backbone network.

17. The apparatus of claim 16, wherein, The processing module is specifically configured to: based on the adjusted one-hot mask matrix associated with the network layer and the gradient returned to the network layer, calculate new gradients of the network layer; based on the new gradients of the network layer, update the parameters of the backbone network. The processing module is specifically configured to: based on the adjusted one-hot mask matrix associated with the network layer, group the channels in the network layer to obtain N channel groups, where N≥2, and different channel groups process different sets of tasks; The gradient modulation algorithm is used to calculate the gradient of each channel included in each of the N channel groups, so as to obtain the weighted weight of each channel group. Based on the weighted weight of each channel group and the gradient returned to the network layer, a new gradient of the network layer is calculated.

18. The apparatus of any of claims 12-17, wherein, The one-hot mask matrix is used for bit multiplication with the parameters of the network layer.

19. The apparatus of any of claims 12-18, wherein, Before the acquisition module acquires the gradient of the T tasks returned to the network layer in the backbone network, the processing module is further configured to: perform logical OR operation on each sub-matrix in the one-hot mask matrix associated with the network layer, as a new sub-matrix associated with each of the T tasks, and perform processing on each of the T tasks based on the new sub-matrix.

20. The apparatus of any of claims 12-18, wherein, The network layer with a weight matrix is multiple, and each network layer with a weight matrix includes shared channels, and at least one network layer with a weight matrix further includes non-shared channels. The i-th non-shared channel in the k-th network layer with a weight matrix is connected with the shared channel in the (k-1)-th network layer with a weight matrix and other channels including the i-th non-shared channel.

21. The apparatus of any of claims 12-20, wherein, The initial value of an element in the one-hot mask matrix is 1.

22. The apparatus of any of claims 12-21, wherein, After the training of the multi-task network is completed, the acquisition module is further configured to acquire user data. The processing module is further configured to perform inference on the user data by using the multi-task network; wherein, during loading of the multi-task network, each one-hot mask matrix is used for bit multiplication with the parameters of the corresponding network layer, and the results obtained by bit multiplication are used for parallel inference on each task in the current inference during the inference process. The processing module is further configured to output the inference results of each task in the current inference.

23. A cluster of computing devices, characterized in that, Each computing device includes a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method of any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-11, wherein the computing device cluster includes at least one computing device.

25. A computer program product comprising instructions, wherein: The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-11, wherein the computing device cluster includes at least one computing device.

Images