Multi-task joint training method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a multi-task joint training method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a plurality of image data samples and a multi-task model; the multi-task model comprises a sharing layer and a plurality of task sub-models; the sharing layer comprises a plurality of task feature layers which are connected in parallel; each task feature layer is connected with one task sub-model, and the task feature layers are used for extracting target task features for the task sub-models; pre-training a sharing layer in the multi-task model based on the plurality of image data samples; and inputting the image data sample into a pre-trained multi-task model, training a plurality of task feature layers and task sub-models in the multi-task model until a preset training stop condition is met, and determining that the training of the multi-task model is completed. By adopting the method, the model training effect of the multi-task model can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multi-task joint training method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence technology, artificial intelligence is applied in more and more fields. Faced with different tasks, the scene data and models used by artificial intelligence technology may be different. Using a unified model to merge different tasks together, a single model has the ability to process multiple tasks, that is, a multi-task model is obtained.

[0003] Traditionally, the training of multi-task models is often done with the help of multiple training samples. That is, the training samples and sample labels corresponding to different tasks are different. Therefore, there are multiple different training samples corresponding to different tasks. The multiple different training samples are input into the multi-task model, and the multi-task model is trained to obtain a trained multi-task model.

[0004] However, in traditional technologies, different tasks focus on different numbers of feature layers and feature depths, and the convergence speeds of different tasks during training are different. Therefore, when training multi-task models based on multiple training samples, the model training effect is poor. Summary of the invention

[0005] Based on this, it is necessary to provide a multi-task joint training method, apparatus, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a multi-task joint training method, the method comprising:

[0007] Acquire multiple image data samples and a multi-task model; the multi-task model comprises a shared layer and multiple task sub-models; the shared layer comprises multiple task feature layers in parallel; each of the task feature layers is connected to one of the task sub-models, and the task feature layer is used to extract target task features for the task sub-model;

[0008] Pre-training a shared layer in the multi-task model based on a plurality of the image data samples;

[0009] The image data sample is input into the pre-trained multi-task model, and the multiple task feature layers and the task sub-models in the multi-task model are trained until a preset training stop condition is met, and it is determined that the training of the multi-task model is completed.

[0010] In one embodiment, the acquiring of multiple image data samples and a multi-task model includes:

[0011] Get a multi-task model;

[0012] Loading multiple task data sets matching the multi-task model;

[0013] For each task data set corresponding to the task, data enhancement and data preprocessing are performed on the task data set based on the data enhancement method corresponding to the task to obtain image data samples corresponding to the task.

[0014] In one embodiment, the pre-training of the shared layer in the multi-task model based on the plurality of image data samples comprises:

[0015] Inputting a plurality of the image data samples into a shared layer in the multi-task model for unsupervised learning training;

[0016] The feature extraction result is outputted through the shared layer of the multi-task model until the pre-training iteration number of the multi-task model reaches a preset iteration number threshold, and the pre-training of the shared layer of the multi-task model is completed.

[0017] In one embodiment, inputting the image data sample into the pre-trained multi-task model, training multiple task feature layers and the task sub-models in the multi-task model until a preset training stop condition is met, and determining that the multi-task model training is completed, includes:

[0018] Inputting the plurality of image data samples into the pre-trained multi-task model, and performing feature extraction on the image data samples through a shared layer in the multi-task model to obtain feature data corresponding to the plurality of tasks;

[0019] Processing the feature data corresponding to the task sub-models through the task sub-models to obtain output results of each task sub-model;

[0020] Based on the loss of the output results of each of the task sub-models, the total loss of the multi-task model is calculated;

[0021] During the update and iteration process of the multi-task model, the weight parameters corresponding to each task sub-model in the multi-task model are adjusted, and the feature data corresponding to the task sub-model is processed by the task sub-model again to obtain the output results of each task sub-model, until the number of iterations of the multi-task model reaches a preset iteration number threshold, it is determined that the multi-task model training is completed.

[0022] In one embodiment, the total loss of the multi-task model is calculated based on the loss of the output results of each of the task sub-models, including:

[0023] For each task sub-model, respectively calculate the branch loss result of the task sub-model;

[0024] The total loss of the multi-task model is obtained by summing up the loss results of each branch and the weight coefficient.

[0025] In one embodiment, the method further comprises:

[0026] Get application data;

[0027] The application data is input into the multi-task model, and the application data is processed by the multi-task model to obtain an output result of a target task type.

[0028] In a second aspect, the present application also provides a multi-task joint training device, the device comprising:

[0029] A first acquisition module is used to acquire multiple image data samples and a multi-task model; the multi-task model includes a shared layer and multiple task sub-models; the shared layer includes multiple task feature layers in parallel; each of the task feature layers is connected to one of the task sub-models, and the task feature layer is used to extract target task features for the task sub-model;

[0030] A pre-training module, used for pre-training a shared layer in the multi-task model based on a plurality of the image data samples;

[0031] A training module is used to input the image data sample into the pre-trained multi-task model, train multiple task feature layers and the task sub-model in the multi-task model until a preset training stop condition is met, and determine that the multi-task model training is completed.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Acquire multiple image data samples and a multi-task model; the multi-task model comprises a shared layer and multiple task sub-models; the shared layer comprises multiple task feature layers in parallel; each of the task feature layers is connected to one of the task sub-models, and the task feature layer is used to extract target task features for the task sub-model;

[0034] Pre-training a shared layer in the multi-task model based on a plurality of the image data samples;

[0035] The image data sample is input into the pre-trained multi-task model, and the multiple task feature layers and the task sub-models in the multi-task model are trained until a preset training stop condition is met, and it is determined that the training of the multi-task model is completed.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0037] Acquire multiple image data samples and a multi-task model; the multi-task model comprises a shared layer and multiple task sub-models; the shared layer comprises multiple task feature layers in parallel; each of the task feature layers is connected to one of the task sub-models, and the task feature layer is used to extract target task features for the task sub-model;

[0038] Pre-training a shared layer in the multi-task model based on a plurality of the image data samples;

[0039] The image data sample is input into the pre-trained multi-task model, and the multiple task feature layers and the task sub-models in the multi-task model are trained until a preset training stop condition is met, and it is determined that the training of the multi-task model is completed.

[0040] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0041] Acquire multiple image data samples and a multi-task model; the multi-task model comprises a shared layer and multiple task sub-models; the shared layer comprises multiple task feature layers in parallel; each of the task feature layers is connected to one of the task sub-models, and the task feature layer is used to extract target task features for the task sub-model;

[0042] Pre-training a shared layer in the multi-task model based on a plurality of the image data samples;

[0043] The image data sample is input into the pre-trained multi-task model, and the multiple task feature layers and the task sub-models in the multi-task model are trained until a preset training stop condition is met, and it is determined that the training of the multi-task model is completed.

[0044] The above-mentioned multi-task joint training method, device, computer equipment, computer-readable storage medium and computer program product obtain image data samples and multi-task models corresponding to multiple tasks; the multi-task model includes a shared layer and multiple task sub-models; the shared layer is pre-trained based on the image data samples; the shared layer includes multiple task feature layers in parallel; each of the task feature layers is used to extract features of the target task in the multi-tasks; the image data samples are input into the pre-trained multi-task model, and the multiple task feature layers and the task sub-models in the multi-task model are independently trained until the preset training stop condition is met, and it is determined that the training of the multi-task model is completed. By adopting this method, the shared layer of the multi-task model is pre-trained during the training process to ensure the general feature expression ability of the model, and by differentiating multiple task feature layers in the shared layer of the multi-task model, each task feature layer pays more attention to the feature extraction of the corresponding task, avoiding the different number of feature layers and feature depths paid attention to by different tasks. Then, each task feature layer and each task sub-model in the multi-task model are independently trained in combination with the image data samples corresponding to the multiple tasks. There is no need to merge the data labels between the tasks, and the model convergence speed between the tasks is balanced to ensure the model training effect of the multi-task model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 1 is a flow chart of a multi-task joint training method in one embodiment;

[0047] Figure 2 A schematic diagram of a flow chart of steps for constructing an image data sample in one embodiment;

[0048] Figure 3 A schematic diagram of a process flow of a shared layer pre-training step in a multi-task model in one embodiment;

[0049] Figure 4 A schematic diagram of a shared layer structure in a multi-task model in one embodiment;

[0050] Figure 5 A schematic diagram of a process flow of a multi-task model training process in one embodiment;

[0051] Figure 6 A schematic diagram of a flow chart of a step of calculating total loss in one embodiment;

[0052] Figure 7 A schematic diagram of a flow chart of a method for applying a multi-task model in one embodiment;

[0053] Figure 8 It is a flowchart of the process of processing each structure of the model during the joint training of the multi-task model in one embodiment;

[0054] Fig. 9 is a structural block diagram of a multi-task joint training device in one embodiment;

[0055] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In one embodiment, Figure 1 As shown, a multi-task joint training method is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0058] Step 102, obtaining multiple image data samples and a multi-task model.

[0059] The multi-task model includes a shared layer and multiple task sub-models. The shared layer includes multiple task feature layers in parallel; each task feature layer is connected to a task sub-model, and the task feature layer is used to extract target task features for the task sub-model.

[0060] In implementation, when training a multi-task model, the terminal obtains an initial multi-task model and collects image training data of different tasks for the multi-task model. Specifically, the multi-task model is a model that combines task models of different tasks for joint training. For example, a multi-task model can include tasks of different functional types for joint training, such as classification tasks, segmentation tasks, target detection tasks, etc. It also supports joint training of tasks of the same type, such as multiple image recognition tasks for different image contents. Therefore, the training data between multiple tasks are different, and the labels carried by the training data are different. A data loader is integrated in the terminal, and the data loader is used to load the data sets of all tasks corresponding to the multi-task model.

[0061] Step 104: pre-train the shared layers in the multi-task model based on multiple image data samples.

[0062] In implementation, a shared layer is included in the multi-task model, and the shared layer is a common part in different task sub-models in the multi-task model. The pre-trained shared layer has the ability to express the common features of multiple tasks. Specifically, multiple image data samples acquired by the terminal are input into the multi-task model, and a pre-trained loss function is set for the shared layer of the multi-task model. Then, the shared layer is trained for unsupervised learning based on the pre-trained loss function. When the pre-training stop condition is met, the pre-trained shared layer is obtained. The pre-trained shared layer has the ability to express the common features of multiple tasks. For example, for a multi-task model that includes classification tasks, segmentation tasks, and target detection tasks for vehicle images, the shared layer of the multi-task model has the ability to express the common features of the vehicle appearance. The terminal pre-trains the shared layer in the multi-task model based on multiple image data samples.

[0063] Step 106, input the image data sample into the pre-trained multi-task model, train multiple task feature layers and task sub-models in the multi-task model, until the preset training stop condition is met, and determine that the multi-task model training is completed.

[0064] In implementation, different tasks in the multi-task model correspond to task sub-models. For example, if the multi-task model includes classification tasks, segmentation tasks, and natural detection tasks, then the multi-task model includes task sub-models corresponding to these three different tasks. In addition, a taskModule is added to the shared layer of the multi-task model, in which a task feature layer is set for each task. Then, multiple task sub-models are connected in parallel through the task feature layer so that each task sub-model pays more attention to the features it needs. In this way, after the pre-training of the shared layer in the multi-task model is completed, the terminal inputs the image data sample into the pre-trained multi-task model, and jointly trains the multiple task feature layers and task sub-models in the multi-task model until the preset training stop conditions are met, and it is determined that the training of the multi-task model is completed. The trained multi-task model can realize the application processing of multiple tasks.

[0065] Among them, the training stop conditions corresponding to the multi-task model include but are not limited to reaching a preset number of training iterations or the model loss reaching a stable state (that is, the loss value fluctuates within a preset range and no longer changes significantly).

[0066] In this embodiment, the shared layer of the multi-task model is pre-trained during the training process to ensure the general feature expression capability of the model, and by differentiating multiple task feature layers in the shared layer of the multi-task model, each task feature layer pays more attention to the feature extraction of the corresponding task, avoiding the different number of feature layers and feature depths paid attention to by different tasks. Then, each task feature layer and each task sub-model in the multi-task model are independently trained in combination with the image data samples corresponding to the multiple tasks. There is no need to merge the data labels between the tasks, and the model convergence speed between the tasks is balanced to ensure the model training effect of the multi-task model.

[0067] In an exemplary embodiment, Figure 2 As shown, the specific processing process of step 102 includes the following steps 202 to 206. Among them:

[0068] Step 202, obtaining a multi-task model.

[0069] In implementation, the terminal obtains a multi-task model. The model framework of the multi-task model can be a machine learning model, a neural network model, a natural language processing model, etc. The specific model framework of the multi-task model can be selected and determined based on the needs of the business application. The embodiment of the present disclosure does not limit the model framework type of the multi-task model.

[0070] Step 204: load multiple task data sets that match the multi-task model.

[0071] In the implementation, the terminal loads multiple task data sets that match the multi-task model through a pre-integrated data loader. Different tasks correspond to different task data sets, which may carry different data labels and have different data volumes.

[0072] Step 206 , for each task data set corresponding to the task, data enhancement and data preprocessing are performed on the task data set based on the data enhancement method corresponding to the task to obtain image data samples corresponding to the task.

[0073] In implementation, after obtaining multiple task data sets, different task data sets select different data enhancement methods for data enhancement. For example, for the task data set of image data, the image data can be enhanced by geometric transformation methods or pixel transformation methods. For example, the geometric transformation method includes image flipping, rotation, cropping, translation, dithering, etc. to enhance the data set; or, the pixel transformation method includes: adding salt and pepper noise, adding Gaussian noise, performing Gaussian blur, adjusting HSV contrast, adjusting brightness, saturation, white balance, etc. In this way, after selecting appropriate data enhancement methods for task data sets corresponding to different tasks for data enhancement, the terminal continues to perform data preprocessing on the task data in each task data set, for example, by data cleaning, screening and data standardization, etc., the task data set is processed to obtain each processed task data set. Each processed task data set constructs the image data samples corresponding to each task. Among them, the data volume of the image data samples corresponding to different tasks in the image data samples can be different, but it is necessary to ensure that the data of each task must participate in the training, that is, each task has a corresponding task data set.

[0074] In this embodiment, by performing different methods of data enhancement and data preprocessing on different tasks in the collected task data set, image data samples corresponding to multi-task joint training are obtained, ensuring that each task has training data that meets the task characteristics, thereby improving the training accuracy of the multi-task model.

[0075] In an exemplary embodiment, Figure 3 As shown, the specific processing process of step 104 includes steps 302 to 304. Among them:

[0076] Step 302: Input multiple image data samples into a shared layer in a multi-task model for unsupervised learning training.

[0077] In the implementation, the terminal inputs multiple image data samples corresponding to the multi-task model into the shared layer in the multi-task model, uses unsupervised learning methods to pre-train the shared layer, and then performs supervised model training downstream according to the specific usage scenario (i.e., the specific tasks included in the multi-task), which will make the model generalization performance and accuracy higher. Among them, the shared layer of the multi-task model is the backbone (backbone network), and its structure can use transform, ViT, resnet and other structures. The disclosed embodiment does not limit the shared layer structure of the multi-task model. Figure 4 As shown, Figure 4The figure is a structural diagram of the shared layer of the multi-task model. When the shared layer is a backbone network, multiple tasks correspond to multiple Block nodes. The multiple Block nodes are used to extract common features from image data samples. At the same time, taskModule modules are added to some layers of the multiple Block nodes. The taskModule modules are added in parallel to the learnable modules of each task in the multi-task model. The taskModule modules can be modules composed of any small operators, for example, FFN (Feed-Forward Network, forward neural network) layer, conv (convolutional layer) + linear (linear layer) + activation (activation layer), etc. In the present disclosure, taskModule modules are selectively added to the target layer in the shared layer. Different tasks focus on different feature layers and feature depths. The taskModule module can solve this problem and allow each task sub-model in the multi-task model to learn to focus on the feature layers it needs.

[0078] Step 304, outputting feature extraction results through the shared layer of the multi-task model until the number of pre-training iterations of the multi-task model reaches a preset iteration number threshold, and the pre-training of the shared layer of the multi-task model is completed.

[0079] In implementation, during the pre-training process, the terminal processes the image data samples through the shared layer of the multi-task model and outputs the feature extraction results. Then, the multi-task model adaptively adjusts the parameters of the shared layer, continues to process the image data samples, and outputs the feature extraction results. Until the number of pre-training iterations of the multi-task model reaches a preset iteration threshold, the shared layer of the multi-task model tends to converge, and the accuracy of the feature extraction results obtained by multiple iterations also reaches a stable state, that is, it fluctuates within a preset numerical range, indicating that the pre-training of the shared layer of the multi-task model is completed.

[0080] In this embodiment, by pre-training the shared layer in the multi-task model, the multi-task model has the ability to express the common features of multiple tasks. Then, the task sub-models downstream of the shared layer are independently trained, which can improve the model generalization ability and accuracy of the multi-task model.

[0081] In an exemplary embodiment, Figure 5 As shown, the specific processing process of step 106 is:

[0082] Step 502: Input multiple image data samples into the pre-trained multi-task model, and perform feature extraction on the image data samples through the shared layer in the multi-task model to obtain feature data corresponding to multiple tasks.

[0083] In implementation, after the pre-training of the shared layer is completed, formal model training is performed on the multi-task model, and multiple image data samples are input into the pre-trained multi-task model. Feature extraction is performed on the image data samples through the shared layer in the multi-task model. Since the shared layer contains taskModule modules, after the general feature extraction, the features of the task itself can be extracted based on the taskModule modules corresponding to different tasks (the task itself has a corresponding feature extraction depth and feature layer). Moreover, since each taskModule module is connected to the corresponding task sub-model, after feature extraction through the taskModule in the shared layer, the feature data extracted from multiple tasks is input into the corresponding task sub-model.

[0084] Optionally, when inputting data into the multi-task model, different image data samples are fed to taskModule modules corresponding to different tasks, and then transmitted to different task sub-models by pre-developing and configuring the relationship between image data samples and related modules and sub-models in the model. Similarly, the configuration interface of the association relationship can be opened to users, and the corresponding association relationship parameters can be adjusted by displaying readable parameter information in the visual interface, so that the training data corresponding to different tasks can be fed to different learning modules and different task sub-models after being input into the multi-task model.

[0085] Step 504: Process the feature data corresponding to the task sub-model through the task sub-model to obtain the output results of each task sub-model.

[0086] In implementation, after being processed by the shared layer, the feature data extracted by the shared layer is transmitted to the task sub-model, and the terminal processes the feature data corresponding to the task sub-model through the task sub-model to obtain the corresponding output result. For example, the corresponding classification model, segmentation model and target detection model output classification results, image segmentation results and target detection and recognition results respectively. Specifically, the task branch contained in the task sub-model in the multi-task model can be a model with a consistent structure or a model structure constructed by itself. For example, the task sub-model for the target detection task in the multi-task model can use the target detection yolov5 model, and its branch can use the original neck layer (the neck layer is usually composed of multiple groups of convolutional layers, pooling layers, fully connected layers and other types of layers) and head layer (the head layer is usually composed of a group of convolutional layers, fully connected layers or other types of layers) of the yolov5 model.

[0087] Step 506: Based on the loss of the output results of each task sub-model, the total loss of the multi-task model is calculated.

[0088] In implementation, based on the output results of each task sub-model, the terminal uses a supervised training method to calculate the result loss of each task sub-model (that is, the loss of each task branch) based on the label data in the image data sample. Specifically, since the image data samples corresponding to each task are different and the data labels are not merged, the terminal only calculates the loss of the corresponding task when calculating the loss. For each task, there may be a category imbalance, and corresponding adjustment methods can also be set for category imbalance. For example, in classification tasks, for example, focal loss (a loss function for classification problems) is used to give higher weights to categories with fewer categories, or to train categories with fewer categories more times. Furthermore, when calculating the loss corresponding to the current task, the losses of other tasks are not calculated or are set to 0. In addition, in response to the problem of different loss convergence speeds between different tasks, the present disclosure also proposes a solution, that is, when loading data, the batch size of tasks that are easy to converge is made smaller, and the batch size of data tasks that are more difficult to converge is made larger. For example, in general, the task convergence speed of target detection tasks is slower than that of classification tasks. Therefore, the batch size of image data samples of target detection tasks is made larger, and the batch size of image data samples of classification tasks is made smaller. In this way, after determining the loss of the output results corresponding to each task sub-model, the loss of each branch task is merged to obtain the total loss of the multi-task model. Model training is performed based on the loss of each task branch from a local perspective and the total loss from a global perspective.

[0089] Step 508, during the update and iteration process of the multi-task model, the weight parameters corresponding to each task sub-model in the multi-task model are adjusted, and the feature data corresponding to the task sub-model is processed again through the task sub-model to obtain the output results of each task sub-model, until the number of iterations of the multi-task model reaches a preset iteration number threshold, it is determined that the multi-task model training is completed.

[0090] In implementation, after obtaining the losses corresponding to each task sub-model of the multi-task model and the total loss, the terminal can adjust the model parameters in the multi-task model based on each loss, and then continue to iterate the model based on the image data samples. In this way, the terminal keeps adjusting the weight parameters corresponding to each task sub-model in the multi-task model during the update and iteration process of the multi-task model, and re-executes the processing of the feature data corresponding to the task sub-model through the task sub-model until the number of iterations of the multi-task model reaches the preset iteration number threshold, the model of the multi-task model converges, and the terminal determines that the multi-task model training is completed.

[0091] In this embodiment, by performing supervised merge training on the multi-task model, there is no need to merge the data labels between the tasks. By using the label data corresponding to the tasks, it is possible to jointly train the task sub-models in the multi-task model, thereby improving the model generalization ability and robustness of the multi-task model.

[0092] In an exemplary embodiment, Figure 6 As shown, the specific processing process of step 506 includes:

[0093] Step 602: for each task sub-model, respectively calculate the branch loss result of the task sub-model.

[0094] In the implementation, the terminal calculates the branch loss results of each task sub-model separately. When calculating the branch loss results corresponding to a task sub-model, the coefficients corresponding to the loss results of other task sub-models are simply set to 0, or the branch loss results of other task sub-models are not calculated, so that each task sub-model can perform independent loss assessment.

[0095] Step 604, calculate the total loss of the multi-task model by summing up the loss results of each branch and the weight coefficient.

[0096] In the implementation, the terminal calculates the total loss of the multi-task model based on the sum of the loss results of each branch and the preset weight coefficient. Specifically, the total loss calculation method is as follows:

[0097] (1)

[0098]

[0099] Among them, in the above formula (1), ( =1, 2, … ) is the weight coefficient of the loss of each task sub-model. By default, the weight of each task branch is the same. Of course, it can also be allocated according to the importance of each task, and it can also be searched out by other algorithms such as genetic algorithms. The corresponding weight coefficient of each task branch. Indicates To ensure that the loss converges at the same speed between different tasks, the weight learning rate of each branch can be different when the multi-task model updates the gradient.

[0100] In this embodiment, by calculating the branch loss results of each task sub-model, there is no need to merge the data labels between the tasks. The label data corresponding to the task is used, which saves the work of merging data between different tasks. The branch loss results of the multi-task sub-models are merged to determine the total model loss, thereby realizing the merged training of the multi-task model and improving the model generalization ability.

[0101] In an exemplary embodiment, Figure 7 As shown, the method also includes:

[0102] Step 701, obtaining application data.

[0103] In implementation, after the multi-task model training is completed, the terminal obtains application data, which can be application data of any pre-trained task, for example, the application data can be application data of the target detection task. It can also be application data corresponding to the image segmentation task, etc. The disclosed embodiment does not limit the type of the application data.

[0104] Step 702: input the application data into the multi-task model, and process the application data through the multi-task model to obtain an output result of the target task type.

[0105] During implementation, the terminal inputs application data into the multi-task model, processes the application data through the multi-task processing capabilities pre-learned by the multi-task model, clarifies the target task type corresponding to the application data, and thereby obtains the output result of the target task type.

[0106] In this embodiment, application data is processed through a pre-trained multi-task model to obtain the output results of the corresponding tasks, thereby realizing the application of the multi-task model. A feature layer is added to the shared layer of the multi-task model, and each feature layer can pay more attention to the characteristics of its own task. Therefore, the output results obtained through processing by the multi-task model are more accurate.

[0107] In an exemplary embodiment, Figure 8 As shown, an example of a specific processing process of a multi-task model joint training method is given, and the example specifically includes:

[0108] Step 801, load the data of all tasks, obtain image data samples and input them into the multi-task model.

[0109] Step 802, pre-training the shared layer backbone in the multi-task model by an unsupervised learning method, and the pre-trained backbone has the ability to express general features.

[0110] Step 803: The input image data samples are transmitted to the corresponding task sub-models respectively, and supervised training is performed on each task sub-model.

[0111] Step 804, calculating the branch loss results of each task sub-model.

[0112] Step 805, summarizing the branch loss results of each task sub-model to obtain the total loss of the multi-task model.

[0113] Repeat steps 803 to 805 until the number of iterations of the multi-task model reaches a preset iteration number threshold, and then determine that the multi-task model training is completed.

[0114] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a multi-task joint training device for implementing the multi-task joint training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multi-task joint training device embodiments provided below can refer to the limitations of the multi-task joint training method above, and will not be repeated here.

[0116] In an exemplary embodiment, Fig. 9 As shown, a multi-task joint training device 900 is provided, comprising: a first acquisition module 901, a pre-training module 902 and a training module 903, wherein:

[0117] The first acquisition module 901 is used to acquire multiple image data samples and a multi-task model; the multi-task model includes a shared layer and multiple task sub-models; the shared layer includes multiple task feature layers in parallel; each task feature layer is connected to a task sub-model, and the task feature layer is used to extract target task features for the task sub-model;

[0118] A pre-training module 902, used for pre-training a shared layer in a multi-task model based on multiple image data samples;

[0119] The training module 903 is used to input the image data samples into the pre-trained multi-task model, train multiple task feature layers and task sub-models in the multi-task model, until the preset training stop condition is met, and determine that the multi-task model training is completed.

[0120] In one embodiment, the first acquisition module 901 is specifically used to acquire a multi-task model;

[0121] Load multiple task datasets that match the multi-task model;

[0122] For the task data set corresponding to each task, data enhancement and data preprocessing are performed on the task data set based on the data enhancement method corresponding to the task to obtain the image data samples corresponding to the task.

[0123] In one embodiment, the pre-training module 902 is specifically used to input multiple image data samples into a shared layer in a multi-task model for unsupervised learning training;

[0124] The feature extraction results are output through the shared layer of the multi-task model until the pre-training iteration number of the multi-task model reaches a preset iteration number threshold, and the pre-training of the shared layer of the multi-task model is completed.

[0125] In one embodiment, the training module 903 is specifically used to input multiple image data samples into a pre-trained multi-task model, and extract features from the image data samples through a shared layer in the multi-task model to obtain feature data corresponding to multiple tasks;

[0126] The feature data corresponding to the task sub-model is processed through the task sub-model to obtain the output results of each task sub-model;

[0127] Based on the loss of the output results of each task sub-model, the total loss of the multi-task model is calculated;

[0128] During the update and iteration process of the multi-task model, the weight parameters corresponding to each task sub-model in the multi-task model are adjusted, and the feature data corresponding to the task sub-model is processed again through the task sub-model to obtain the output results of each task sub-model. When the number of iterations of the multi-task model reaches the preset iteration number threshold, it is determined that the multi-task model training is completed.

[0129] In one embodiment, the training module 903 is specifically used to calculate the branch loss result of the task sub-model for each task sub-model;

[0130] The total loss of the multi-task model is obtained by summing up the loss results of each branch and the weight coefficient.

[0131] In one embodiment, the apparatus 900 further includes:

[0132] A second acquisition module is used to acquire application data;

[0133] The data processing module is used to input application data into the multi-task model, process the application data through the multi-task model, and obtain the output result of the target task type.

[0134] Each module in the above multi-task joint training device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0135] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a multi-task joint training method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0136] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0138] Acquire multiple image data samples and a multi-task model; the multi-task model includes a shared layer and multiple task sub-models; the shared layer includes multiple task feature layers in parallel; each task feature layer is connected to a task sub-model, and the task feature layer is used to extract target task features for the task sub-model;

[0139] Pre-train the shared layers in the multi-task model based on multiple image data samples;

[0140] The image data samples are input into the pre-trained multi-task model, and multiple task feature layers and task sub-models in the multi-task model are trained until the preset training stop conditions are met, and it is determined that the multi-task model training is completed.

[0141] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0142] Get a multi-task model;

[0143] Load multiple task datasets that match the multi-task model;

[0144] For the task data set corresponding to each task, data enhancement and data preprocessing are performed on the task data set based on the data enhancement method corresponding to the task to obtain the image data samples corresponding to the task.

[0145] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0146] Input multiple image data samples into the shared layer of the multi-task model for unsupervised learning training;

[0147] The feature extraction results are output through the shared layer of the multi-task model until the pre-training iteration number of the multi-task model reaches a preset iteration number threshold, and the pre-training of the shared layer of the multi-task model is completed.

[0148] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0149] Input multiple image data samples into the pre-trained multi-task model, and extract features from the image data samples through the shared layer in the multi-task model to obtain feature data corresponding to multiple tasks;

[0150] The feature data corresponding to the task sub-model is processed through the task sub-model to obtain the output results of each task sub-model;

[0151] Based on the loss of the output results of each task sub-model, the total loss of the multi-task model is calculated;

[0152] During the update and iteration process of the multi-task model, the weight parameters corresponding to each task sub-model in the multi-task model are adjusted, and the feature data corresponding to the task sub-model is processed again through the task sub-model to obtain the output results of each task sub-model. When the number of iterations of the multi-task model reaches the preset iteration number threshold, it is determined that the multi-task model training is completed.

[0153] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0154] For each task sub-model, calculate the branch loss results of the task sub-model respectively;

[0155] The total loss of the multi-task model is obtained by summing up the loss results of each branch and the weight coefficient.

[0156] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0157] Get application data;

[0158] The application data is input into the multi-task model, and the application data is processed by the multi-task model to obtain the output result of the target task type.

[0159] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0160] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0161] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0163] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-task joint training method, characterized in that: The method comprises: Acquire multiple image data samples and a multi-task model; the multi-task model comprises a shared layer and multiple task sub-models; the shared layer comprises multiple task feature layers in parallel; each of the task feature layers is connected to one of the task sub-models, and the task feature layer is used to extract target task features for the task sub-model; Pre-training a shared layer in the multi-task model based on a plurality of the image data samples; The image data sample is input into the pre-trained multi-task model, and the multiple task feature layers and the task sub-models in the multi-task model are trained until a preset training stop condition is met, and it is determined that the training of the multi-task model is completed.

2. The method according to claim 1, characterized in that The obtaining of multiple image data samples and a multi-task model includes: Get a multi-task model; Loading multiple task data sets matching the multi-task model; For each task data set corresponding to the task, data enhancement and data preprocessing are performed on the task data set based on the data enhancement method corresponding to the task to obtain image data samples corresponding to the task.

3. The method according to claim 1, characterized in that The pre-training of the shared layer in the multi-task model based on the plurality of image data samples comprises: Inputting a plurality of the image data samples into a shared layer in the multi-task model for unsupervised learning training; The feature extraction result is outputted through the shared layer of the multi-task model until the pre-training iteration number of the multi-task model reaches a preset iteration number threshold, and the pre-training of the shared layer of the multi-task model is completed.

4. The method according to claim 1, characterized in that: The step of inputting the image data sample into the pre-trained multi-task model, training the multiple task feature layers and the task sub-models in the multi-task model until a preset training stop condition is met, and determining that the multi-task model training is completed includes: Inputting the plurality of image data samples into the pre-trained multi-task model, and performing feature extraction on the image data samples through a shared layer in the multi-task model to obtain feature data corresponding to the plurality of tasks; Processing the feature data corresponding to the task sub-models through the task sub-models to obtain output results of each task sub-model; Based on the loss of the output results of each of the task sub-models, the total loss of the multi-task model is calculated; During the update and iteration process of the multi-task model, the weight parameters corresponding to each task sub-model in the multi-task model are adjusted, and the feature data corresponding to the task sub-model is processed by the task sub-model again to obtain the output results of each task sub-model, until the number of iterations of the multi-task model reaches a preset iteration number threshold, it is determined that the multi-task model training is completed.

5. The method according to claim 4, characterized in that The total loss of the multi-task model is calculated based on the loss of the output results of each of the task sub-models, including: For each task sub-model, respectively calculate the branch loss result of the task sub-model; The total loss of the multi-task model is obtained by summing up the loss results of each branch and the weight coefficient.

6. The method according to claim 1, characterized in that The method further comprises: Get application data; The application data is input into the multi-task model, and the application data is processed by the multi-task model to obtain an output result of a target task type.

7. A multi-task joint training device, characterized in that: The device comprises: An acquisition module is used to acquire multiple image data samples and a multi-task model; the multi-task model includes a shared layer and multiple task sub-models; the shared layer includes multiple task feature layers in parallel; each of the task feature layers is connected to one of the task sub-models, and the task feature layer is used to extract target task features for the task sub-model; A pre-training module, used for pre-training a shared layer in the multi-task model based on a plurality of the image data samples; A training module is used to input the image data sample into the pre-trained multi-task model, train multiple task feature layers and the task sub-model in the multi-task model until a preset training stop condition is met, and determine that the multi-task model training is completed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images