Multi-task model training method and device, labeling method and device and storage medium
By calculating the comprehensive loss in multi-task model training and performing a backpropagation update parameter, the problem of seesaw effect in multi-task model training is solved, and the recognition accuracy of the model on different tasks is improved.
Patent Information
- Application Number
- CN202411084862.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-22
AI Technical Summary
In the existing multi-task model training method, the loss value corresponding to each output result will be feedback and updated to the shared parameters, resulting in a seesaw effect of multi-task models with poor correlation. The trained model has a large difference in the recognition accuracy of different tasks.
A multi-task model training method is adopted. After obtaining multiple data, the comprehensive loss is calculated based on multiple data of a single task type, and backpropagation is performed to update the parameters of the shared network module and task head module, and after judging the training conditions, Freeze the shared network module to avoid excessive update of shared parameters during backpropagation.
This method reduces the difference in recognition accuracy of multi-task models on different tasks, reduces the impact of seesaw effect, and improves the overall recognition accuracy of multi-task models.
Smart Images

Figure CN119004105A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multi-task model training, and in particular to a multi-task model training method and device, a labeling method and device, and a storage medium. Background Art
[0002] At present, the processing of model data is becoming more and more important. However, manually labeling this data is a tedious and time-consuming task. Most single-task model training methods are only applicable to a single model, and for different task types, model training needs to be repeated, which greatly increases the cost of use.
[0003] Multi-task learning (MTL) can solve multiple learning tasks at the same time. Compared with training models separately, MTL uses the commonalities and differences between tasks to improve the learning efficiency of specific task models.
[0004] Merge multiple tasks into one model to reduce the maintenance cost of multiple different task models. By sharing embedding parameters, the parameter scale can be reduced, resources can be saved, and multiple task results can be obtained with only one model request, reducing online latency.
[0005] However, in the existing multi-task model training methods, the loss value corresponding to each output result will be fed back to update the shared parameters once. For multi-task models with poor correlation, the recognition accuracy of the trained model for different tasks varies greatly, which will produce a serious seesaw effect. That is, the trained multi-task model has a good recognition accuracy for data of task type A, but a poor accuracy for data of task type B. Summary of the invention
[0006] In view of the defects existing in the prior art, the purpose of the present invention is to provide a multi-task model training method and device, a labeling method and device, and a storage medium, which can solve the problem in the prior art that the loss value corresponding to each output result is fed back to update the shared parameters once, which will produce a serious seesaw effect for multi-task models with poor correlation.
[0007] In order to achieve the above purpose, the technical solution adopted by the present invention is:
[0008] In a first aspect, the present invention provides a multi-task model training method, characterized in that it comprises the following steps:
[0009] Get multiple data;
[0010] For each data, the output result is obtained after being processed by the shared network module with fixed parameters and the corresponding task head module, and the loss is calculated between the output result and the corresponding label data;
[0011] Based on multiple losses corresponding to multiple data of a single task type, the comprehensive loss of the task type is obtained, and a back propagation is performed based on the comprehensive loss to update the parameters of the shared network module and the task head module;
[0012] Determine whether the training end conditions are met;
[0013] If not, return to get multiple data; if yes, freeze the parameters of the shared network module.
[0014] In some optional solutions, when acquiring multiple data, data of all task types are acquired at one time, and at least two data of each task type are acquired;
[0015] Each data is processed by a shared network module with fixed parameters and a corresponding task head module, and the loss is calculated;
[0016] The comprehensive loss of each task type is back-propagated in turn to update the parameters of the shared network module and the task head module.
[0017] In some optional schemes, data of all task types are obtained at one time, and when the loss of the data is calculated, the data of each task type is processed in a loop in turn, and the number of loops is set based on the amount of data of all task types.
[0018] In some optional solutions, when acquiring multiple data, data of one task type is acquired each time;
[0019] For each data of this task type, the shared network module and the corresponding task header module are processed to obtain the output result, and the loss is calculated;
[0020] Back propagation is performed based on the comprehensive loss of this task type to update the parameters of the shared network module and the task head module.
[0021] In some optional schemes, the accuracy of the training model corresponding to all task type data sampled once and the accuracy of the training model corresponding to one task type data sampled each time are obtained, and the training model with the highest accuracy is selected as the final training model.
[0022] In some optional solutions, after the step of freezing the parameters of the shared network module, the method further includes:
[0023] Get multiple data;
[0024] For each data, the output result is obtained after being processed by the shared network module with frozen parameters and the task head module with fixed corresponding parameters, and the loss is calculated for the output result and the corresponding label data;
[0025] Based on multiple losses corresponding to multiple data of a single task type, the comprehensive loss of the task type is obtained, and a back propagation is performed based on the comprehensive loss to update the parameters of the corresponding task head module;
[0026] Determine whether the training end conditions are met;
[0027] If not, return to get more data; if so, end the training.
[0028] In some optional schemes, the vector sum of multiple losses corresponding to multiple data of a single task type is used as the comprehensive loss of the task type.
[0029] In some optional solutions, when obtaining multiple data, data corresponding to the weight ratio is obtained according to the weight of each task type.
[0030] In some optional schemes, before obtaining multiple data corresponding to each task type, a static graph of the multi-task flow is pre-generated, and the static graph includes the storage addresses of the shared network module and each task header module, and the order of data processing of each task type. When executing each task, the shared network module and the corresponding task header module are called based on the disk address in the static graph.
[0031] In a second aspect, the present invention further provides a multi-task model training device, comprising:
[0032] A data acquisition module, which is used to acquire multiple data;
[0033] The calculation module is used to obtain the output result after processing each data through the shared network module with fixed parameters and the corresponding task head module, and calculate the loss of the output result and the corresponding label data;
[0034] A parameter updating module, which is used to obtain a comprehensive loss of a single task type based on multiple losses corresponding to multiple data of the task type, and to perform a back propagation based on the comprehensive loss to update the parameters of the shared network module and the task head module;
[0035] The judgment module is used to judge whether the training end condition is met; if not, it returns to obtain multiple data; if so, it freezes the parameters of the shared network module.
[0036] In a third aspect, the present invention also provides a readable storage medium, on which a multi-task model training program is stored, wherein when the multi-task model training program is executed by a processor, the steps of the above-mentioned multi-task model training method are implemented.
[0037] In a fourth aspect, the present invention further provides a multi-task labeling method, comprising the following steps:
[0038] Input the data to be annotated;
[0039] Processing the data using the multi-task model trained by the multi-task model training method;
[0040] Output the labeling results corresponding to each task head module.
[0041] In a fifth aspect, the present invention further provides a multi-task labeling device, comprising:
[0042] An input module, which is used to input data to be annotated;
[0043] A processing module, which is used to process data using the multi-task model trained by the multi-task model training method;
[0044] The output module is used to output the labeling results corresponding to each task head module.
[0045] In a sixth aspect, the present invention further provides a readable storage medium, on which a multi-task labeling program is stored, wherein when the multi-task labeling program is executed by a processor, the steps of the multi-task labeling method described above are implemented.
[0046] Compared with the prior art, the advantages of the present invention are: this scheme obtains a comprehensive loss based on the loss of multiple data corresponding to each task type, performs a back propagation for each comprehensive loss, and updates the parameters of the shared network module and the task head module. For multi-task models with poor correlation, the impact of the serious seesaw effect when sharing parameters can be reduced. In this scheme, the shared network module is updated each time the back propagation is performed, so this method can reduce the impact of the "seesaw effect" on each task type. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0048] Figure 1 A flowchart of a shared network module training according to an embodiment of the present invention;
[0049] Figure 2 This is a flow chart of data processing during multi-task model training in an embodiment of the present invention;
[0050] Figure 3 This is a flowchart of the task head module training in an embodiment of the present invention;
[0051] Figure 4 is a flow chart of a multi-task labeling method according to an embodiment of the present invention;
[0052] Figure 5 A flowchart for searching for an instantiation module in an embodiment of the present invention;
[0053] Figure 6 The figure is a flowchart of multi-task model node management in an embodiment of the present invention. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0055] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0056] like Figure 1 and Figure 2 As shown, on the one hand, the present invention provides a multi-task model training method, comprising the following steps:
[0057] S1: Get multiple data.
[0058] In this example, a batch of data is obtained each time, and the batch of data includes data of multiple task types or includes data of one task type. In both methods, for a batch of data, each task type corresponds to multiple data.
[0059] In some optional embodiments, multiple data are obtained, and data corresponding to the weight ratio is obtained according to the weight of each task type.
[0060] For example, if the multi-task model includes task type A, task type B, and task type C, and the corresponding weights are 4:4:2, then when sampling data, task type A, task type B, and task type C use the data loader to sample a batch of data from the original data in a ratio of 4:4:2. When sampling data, ensure that multiple data can be sampled for each data type.
[0061] In this example, the data is a combination of a picture and description information of the picture task name. The picture task name description information is used to input the data into the processing flow of the corresponding task type. The task types include traffic light detection tasks, lane line detection tasks, pedestrian detection tasks, non-motor vehicle detection tasks, and motor vehicle detection tasks.
[0062] The weight of each task type can be set by the user according to their own needs. By manually controlling the weight of the task type, the influence of each task type in the training process can be affected. Generally, the weight of the task type that positively affects the model training results is set larger.
[0063] In some optional embodiments, according to the weight of each task type, when sampling data corresponding to the weight ratio, the data of all task types are sampled at once or the data of one task type is sampled each time.
[0064] When sampling data, if you sample data of all task types at once, after each data update, sample data, that is, when a batch of data is requested, a list of all task types is returned. The data loader samples data of all task types from the original data based on the list of each task type. The expanded form of sampling data of all task types at once, that is, "Sampleall", is as follows:
[0065] output_unions=[["task1"*factor1,"task2"*factor2,...]]
[0066] Among them, output_unions represents the data set, task1 represents the first task type, and factor1 represents the weight of the first task type.
[0067] If the data of one task type is sampled each time, the data is sampled after each data update, that is, when a batch of data is requested, only a list of one task type is returned. The data loader samples the data of all task types from the original data based on the task type list; each time the data of one task type is sampled, the expanded form of "Sample one" is as follows:
[0068] output_unions=[["task1"]*factor1,["task2"]*factor2,...].
[0069] Among them, unions_unions represents the dataset, task1 represents the first task type, and factor1 represents the weight of the first task type.
[0070] The data sampling part is mainly implemented and executed by the data iterator (Multitask Loader Iterator, MLI). The set parameters are passed to the initialized data loaders (Dataloader) of each task, and the key values of the related tasks are checked internally. Finally, the corresponding task name and the batch data sampled in the current loader are packaged and output together. The packaging format is similar to: (task_name, batch_data), task_name represents the task name, and batch_data represents a batch of multiple data.
[0071] S2: For each data, the output result is obtained after being processed by the shared network module with fixed parameters and the corresponding task head module, and the loss is calculated for the output result and the corresponding label data.
[0072] In this scheme, if data of all task types are sampled at one time, the data of each task type is processed by the shared network module and the corresponding task header module according to the set rules, and the output results are processed with the corresponding label data for loss calculation to obtain multiple losses for all task types, and each task type corresponds to multiple losses.
[0073] If data of one task type is sampled each time, then for each data of this task type, the output result is obtained after being processed by the shared network module and the corresponding task header module, and the loss is calculated by combining the output result with the corresponding label data to obtain multiple losses corresponding to one task type.
[0074] In some optional embodiments, the shared network module includes a backbone network unit and a neck network unit. The backbone network unit is used to extract preliminary feature data, mainly to perform preliminary feature extraction on the input data, capture the low / mid-level features of the input data, and build an abstract representation of the input data.
[0075] Between the shared network module with fixed parameters and the corresponding task head module, the neck network module with fixed parameters is also used to further compress the feature representation of the features extracted from the shared network module, fuse multi-scale information or perform feature enhancement. It can be understood as further analyzing the image / text features to extract image details and semantic information. The output of this part is generally a feature map after further processing, which has higher abstraction and expression ability.
[0076] Finally, the data extracted by the neck will be output to the head part. In the model structure of this part, there are usually multiple heads, each of which corresponds to a different task type, such as: vehicle detection task, traffic light detection task, lane line detection task, pedestrian and non-motor vehicle detection task, motor vehicle detection task, etc. The specific structure is closely related to the specific task. After executing the task specified by the head part, the structure will map the features to the output space.
[0077] In this example, the backbone network unit is a neural network including convolutional layers and pooling layers. The backbone network unit uses a neural network with convolutional layers and pooling layers to improve the efficiency of feature extraction, improve computational efficiency, enhance model generalization ability, and control overfitting.
[0078] S3: Based on the multiple losses corresponding to multiple data of a single task type, the comprehensive loss of the task type is obtained, and a back propagation is performed based on the comprehensive loss to update the parameters of the shared network module and the task head module.
[0079] In this scheme, two different back-propagation update methods are provided based on two different data sampling methods.
[0080] Update method A: When acquiring multiple data, acquire data of all task types at once, and acquire at least two data of each task type; each data is processed by a shared network module with fixed parameters and a corresponding task head module, and the loss is calculated; the comprehensive loss of each task type is back-propagated in turn, and the parameters of the shared network module and the task head module are updated.
[0081] Specifically, a batch of data including all task types is obtained at a time, and multiple data corresponding to each task type are obtained. A shared network module and a task head module with fixed parameters are used to process each data in the batch of data. Multiple losses are obtained for each task type. Based on the multiple losses of each task type, a comprehensive loss of the task type is obtained. In this example, if the multi-task model includes task type A, task type B, and task type C, three comprehensive losses are obtained. The three comprehensive losses are back-propagated in turn to update the parameters of the shared network module and the task head module.
[0082] In this example, data of all task types are obtained at once. When processing data, the amount of data of each task type may be inconsistent. For example, the data corresponding to the weight ratio is obtained according to the set weight, and the weight of each task type is different. Since the parameters of the shared network module and the task head module are fixed at this time, the processing order does not affect the processing results, but for the convenience of processing when using software in this method. The following method is used for processing:
[0083] When calculating the loss of data, the data of each task type is processed in turn in a loop, and the number of loops is set based on the amount of data of all task types.
[0084] Specifically, based on the amount of data for all task types, the number of data processing cycles for all task types is set, and the data of each task type is processed by the shared network module and the corresponding task header module in turn to obtain the output result. The loss is calculated for the output result and the corresponding label data until the set number of data processing cycles is completed. The comprehensive loss of each task type is back-propagated in turn, and the parameters of the shared network module and the task header module are updated.
[0085] Since the amount of data for each task type may be inconsistent, there may be a situation where each task type is executed a different number of times. The number of loops is set based on the amount of data for all task types. In this example, the number of loops is set to the number with the largest amount of data for all task types or the number with the smallest amount of data for all task types.
[0086] Specifically in the settings, when the number of loops is set to "min_size", that is, the minimum amount of data for all task types, the program will control the end of the entire data processing process based on the completion of the traversal of the smaller (fewer return times) loader; and when the number of loops is set to "max_size", that is, the maximum amount of data for all task types, the program will control the end of the entire data processing process based on the completion of the traversal of the larger (more return times) loader. When one or more loaders are sampled to the end, if the data processing process still does not meet the end condition, it will loop to the beginning of the current loader to reprocess the data until the end condition is met.
[0087] For example, the multi-task model needs to complete task type A, task type B, and task type C, and the corresponding weights are 4:4:2.
[0088] If the number of loops is set to the number with the largest amount of data for all task types, the number of loops is 4. When a batch of data is obtained, based on the task names of task types A, B and C, the data loader is used to sample 4 copies of data corresponding to task type A, 4 copies of data corresponding to task type B and 2 copies of data corresponding to task type C from the original data. When the shared network module and the corresponding task header module are used to process this batch of data, the shared network module first processes one of the 4 copies of data corresponding to task type A, and then processes it through the corresponding task header module A to obtain an output result corresponding to task type A. The shared network module processes one of the 4 copies of data corresponding to task type B, and then After being processed by the corresponding task header module B, an output result corresponding to task type B is obtained. The shared network module processes one of the two data corresponding to task type C, and then processes it through the corresponding task header module C to obtain an output result corresponding to task type C; in this way, in the next cycle, other data of task type A, task type B and task type C are processed. In the third round of data processing, after processing the third data of task type B, the shared network module again processes one of the two data corresponding to task type C, and then processes it through the corresponding task header module C to obtain an output result corresponding to task type C; the fourth round of data is processed in the same way.
[0089] When the number of cycles is set to the number with the largest data volume of all task types, the loss of each output result of the task head module is calculated with the corresponding label data. After obtaining the four losses corresponding to task head module A, the four losses corresponding to task head module B, and the four losses corresponding to task head module C, the comprehensive loss obtained by processing the four loss values corresponding to task head module A is back-propagated "once", and the parameters of the shared network module and task head module A are updated. Then, the comprehensive loss obtained by processing the four loss values corresponding to task head module B is back-propagated "once", and the parameters of the shared network module and task head module B are updated. Then, the comprehensive loss obtained by processing the four loss values corresponding to task head module C is back-propagated "once", and the parameters of the shared network module and task head module C are updated, so as to avoid the "seesaw effect".
[0090] If the number of loops is set to the minimum number of data for all task types, the number of loops is 2. When sampling a batch of data, task type A, task type B, and task type C use the data loader to sample 4 data corresponding to task type A, 4 data corresponding to task type B, and 2 data corresponding to task type C from the original data. When the shared network module and the corresponding task header module are used to process this batch of data, the shared network module first processes one of the 4 data corresponding to task type A, and then processes it through the corresponding task header module A to obtain an output result corresponding to task type A. The shared network module processes one of the 4 data corresponding to task type B, and then processes it through the corresponding task header module B to obtain an output result corresponding to task type B. The shared network module processes one of the 2 data corresponding to task type C, and then processes it through the corresponding task header module C to obtain an output result corresponding to task type C. Then, the next loop is performed, and the execution ends after two loops.
[0091] When the number of cycles is set to the minimum number of data for all task types, after obtaining the two losses corresponding to task head module A, the two losses corresponding to task head module B, and the two losses corresponding to task head module C, the comprehensive loss obtained by processing the two loss values corresponding to task head module A is back-propagated "once", and the parameters of the shared network module and task head module A are updated. Then, the comprehensive loss obtained by processing the two loss values corresponding to task head module B is back-propagated "once", and the parameters of the shared network module and task head module B are updated. Then, the comprehensive loss obtained by processing the two loss values corresponding to task head module C is back-propagated "once", and the parameters of the shared network module and task head module C are updated, so as to avoid the "seesaw effect".
[0092] In the embodiment, the above two specific ways of setting the number of cycles are provided, and other numbers of cycles may also be used in practical applications. In addition, when training the model, if a training method of sampling data of all task types at once is selected, multiple numbers of cycles may be set, and corresponding training models may be obtained, and then the training models may be tested and compared, and a training model with a high accuracy rate may be selected as the final training model of the training method of sampling data of all task types at once.
[0093] B: When acquiring multiple data, acquire data of one task type each time; for each data of this task type, obtain the output result after processing by the shared network module and the corresponding task head module, and perform loss calculation; perform back propagation based on the comprehensive loss of this task type, and update the parameters of the shared network module and the task head module.
[0094] Then, when obtaining multiple data in the next batch, obtain multiple data corresponding to the next task type, and perform back propagation based on the comprehensive loss of the next task type until the comprehensive loss of all task types is completed and back propagated. Then obtain the data of the first task type until the training conditions are met.
[0095] In this embodiment, the data of one task type is sampled each time. Take the multi-task model that needs to complete task type A, task type B and task type C as an example, and the corresponding weights are 4:4:2. First, task type A uses the data loader to sample 4 copies of data corresponding to task type A from the original data, and the shared network module processes one of the 4 copies of data corresponding to task type A, and then processes it through the corresponding task head module A to obtain an output result corresponding to task type A, and calculate the loss of the output result and the corresponding label data. In this way, the four losses corresponding to the task head module A are obtained, and the comprehensive loss obtained by processing the 4 loss values corresponding to task type A is back-propagated; then the 4 data corresponding to task type B are obtained, and the comprehensive loss obtained by processing the 4 loss values corresponding to task type B is back-propagated, and finally the 2 data corresponding to task type C are obtained, and the comprehensive loss obtained by processing the 2 loss values corresponding to task type C is back-propagated.
[0096] In addition, when back-propagation updates the shared network module and the task head module, the parameters of the neck network module are also updated.
[0097] In some optional embodiments, the accuracy of the training model corresponding to all task type data sampled once and the accuracy of the training model corresponding to one task type data sampled each time are obtained respectively, and the training model with the highest accuracy is selected as the final training model.
[0098] In specific applications, when training the model, the same original data and corresponding label data are used to train the model by sampling all task type data once and sampling one task type data each time. The two trained models are tested and compared, and the training model with higher accuracy is selected as the final training model.
[0099] S4: Determine whether the training end condition is met. If not, return to obtain multiple data corresponding to each task type; if so, freeze the parameters of the shared network module.
[0100] In this embodiment, the training end condition is to set the number of training times, that is, after the shared network module and the task head module are updated to set the number of training times, it is determined that the training of the model is completed.
[0101] By back-propagating the comprehensive loss values corresponding to each task head module "at one time" and updating the parameters of the shared network module and the task head module, the impact of the serious seesaw effect when sharing parameters can be reduced for multi-task models with poor correlation. In this scheme, the shared network module is updated each time back-propagation is performed. Therefore, this method can reduce the impact of the "seesaw effect" on all task types.
[0102] It should be noted that since the parameters of the shared network module and the neck network module will stabilize quickly in multi-task training, the parameters of these two parts will be frozen in the later stage of multi-task training, that is, only the task head module of each task will be trained to speed up the reasoning speed, ensure the calculation effect, and reduce resource usage.
[0103] like Figure 3 As shown, S5: after the step of freezing the parameters of the shared network module, it also includes: obtaining multiple data; for each data, obtaining an output result after processing by the shared network module with frozen parameters and the task head module with fixed corresponding parameters, and calculating the loss of the output result and the corresponding label data; obtaining the comprehensive loss of the task type based on multiple losses corresponding to multiple data of a single task type, and performing a back propagation based on the comprehensive loss to update the parameters of the corresponding task head module; judging whether the training end condition is met; if not, returning to obtain multiple data; if so, ending the training.
[0104] In this embodiment, the training and updating steps for the parameters of the task head module are basically the same as the steps S1-S4, and the data of all task types can be sampled at one time or the data of one task type can be sampled each time. For each data, after passing through the shared network module with fixed parameters, the output result is obtained after being processed by the corresponding task head module, and the loss of the output result and the corresponding label data is calculated. The comprehensive loss is obtained according to the multiple losses of the single task type data in each batch of data, and then the comprehensive loss is "one-time" back-propagated, and only the parameters of the task head module are updated. After the parameters of each task head module are trained, the trained multi-task model is obtained.
[0105] In addition, this scheme uses the comprehensive loss of multiple losses of a single task type to update the parameters of the shared network module and / or the task head module. Compared with the back propagation after accumulating the losses of multiple task types, it has the following advantages: it can avoid the problem of non-linear combination of different loss functions corresponding to multiple task types, is more flexible, and avoid the problem of gradient update direction conflict caused by simply adding loss values between different task types.
[0106] Similarly, by back-propagating the comprehensive loss obtained from multiple losses corresponding to each task head module "at once" and updating the parameters of the task head module, the "seesaw effect" caused by accidental deviations in training data can be reduced. Freezing the parameters of the shared network model and only training and updating the parameters of each task head model can improve training efficiency.
[0107] In addition, in this example, the parameters of the task head module are trained in the same manner as the shared network parameter training method, so that better task head module parameters can be obtained. In other embodiments, other methods are used to train the parameters of the task head module, which does not affect the core concept of this solution and can also solve the problems existing in the prior art.
[0108] Preferably, the vector sum of multiple losses corresponding to multiple data of a single task type is used as the comprehensive loss of the task type.
[0109] In this example, the comprehensive loss of the task type is back-propagated in sequence, that is, the comprehensive loss values corresponding to each task head module are back-propagated "at one time", which means that for a certain task type, the loss values corresponding to each data of the task type are recorded in a list, and during back-propagation, the loss value vector sum of the list is used instead of the loss value during normal training for back-propagation backward. In other embodiments, the loss values corresponding to each data of the task type are processed and then fed back. For example, the loss values that deviate too much are deleted, and then the vector sum of the remaining loss values is back-propagated. The above methods can reduce the excessive impact of the loss value caused by a single data on the parameters of the shared network module and / or the task head module, so as to reduce the seesaw effect.
[0110] like Figure 5 and Figure 6 As shown, in some optional embodiments, before obtaining multiple data corresponding to each task type, a static graph of the multi-task flow is pre-generated, and the static graph includes the storage addresses of the shared network module and each task header module, and the order of data processing of each task type data. When executing each task, the shared network module and the corresponding task header module are called based on the disk address in the static graph.
[0111] In this embodiment, the multi-task model is instantiated and stored in the disk. When performing data processing, different computing modules are selected based on the task type corresponding to the input data, and the modules of different tasks are selectively loaded into the memory according to the storage address.
[0112] Specifically, the instantiated shared network module task header module is stored in the disk, and the model parameters are configured when the multi-task model is started. By integrating the name, parameters, definition and other information of the instantiated shared network module task header module, and saving the storage address of the shared network module task header module in the environment variable in advance, the "lazy loading" mode is realized. This mode will only instantiate the model node and its related components when the specific module of the specified task is specifically called. For small models with fewer nodes involved in one infer process, this calling method can avoid loading a large amount of resources at the startup stage, save the use of resources such as memory and processor, and execute the functions required by the program with lower resource occupancy and faster response speed, and reduce unnecessary performance loss at runtime. However, it should be noted that this method is not suitable for model scenarios with frequent calls to more head nodes and multiple reasoning operations. Because the "lazy loading" mode can delay resource loading or object creation, it is still necessary to explicitly instantiate the head node specified by the model when it is first accessed. Since this part is not instantiated at the beginning of program execution, it is necessary to address and establish the corresponding node object during execution, which intuitively reduces the speed.
[0113] After solving the model calling problem, it is necessary to effectively manage the nodes of the task head modules (head) of multiple task types. In fact, for networks of various task types (such as pedestrian detection and vehicle detection in detection tasks), their structures are mostly very similar. In multi-task training scenarios, it is necessary to accurately call each head node and ensure that each batch of data can be dynamically transferred to different heads in the corresponding process.
[0114] Based on the above usage scenarios, such as Figure 3 As shown in the figure, when starting to run the multi-task model, a static graph of the entire multi-task flow is pre-generated. The static graph includes the disk addresses of the shared network module and each task head module, as well as the order of data processing for each task type. The static graph is generated according to the configuration file. The configuration file contains the settings of the entire training process, including the task type, the number of layers of the model, the number of training generations, the data storage address and other model internal parameters and execution parameters.
[0115] The specific static graph generation process is as follows: extract the common nodes of each task. If not specified, the common nodes in multiple tasks usually refer to the backbone and neck modules, that is, the shared network module. For the remaining heads of different tasks in different scenarios, each head will be marked and saved as a dictionary in the current execution memory. Subsequently, the entire model is instantiated in the "lazy loading" method mentioned above, and its content includes the common backbone and neck modules, as well as the head modules of different task scenarios.
[0116] In the specific static graph creation process, the inherent properties and methods of different network frameworks are dynamically inherited, and the extracted public nodes are set as internal properties of the static graph that can be called; then, in the instantiation method of the static graph, the head nodes of each task that may be contained therein are determined, marked and treated in the same way as the public nodes, so that they can be used as internal properties that can be called in the static graph object after instantiation.
[0117] It is also important to note that when marking and saving the non-public node head, the forward calculation and other methods of the network corresponding to the head will also be saved in the memory area mentioned above as a dictionary to store the head. The purpose is to correspond to the independently defined function methods when performing forward calculations for different tasks. This type of method will select the corresponding node from the internal attributes of the current object for pre-processing, feature extraction, feature conversion, post-processing and other steps according to the set process. Since all nodes (including public nodes backbone, neck and non-public node head) have been saved in the internal attributes during the previous operation, they can be called directly.
[0118] like Figure 6 As shown in the figure, the specific processing process is as follows: when starting the multi-task model, a static graph of the multi-task flow is generated in advance. When executing data processing, the task name of the input data is first checked, and the shared network module and the corresponding task header module are called according to the order of data processing of the task type and the storage address of the required module to process the data. After processing the data, the called module will continue to be saved in the running memory to process this type of data.
[0119] In a second aspect, the present invention further provides a multi-task training device, comprising: a data acquisition module, a calculation module, a parameter updating module and a judgment module.
[0120] The data acquisition module is used to acquire multiple data; the calculation module is used to obtain the output result after processing each data through the shared network module with fixed parameters and the corresponding task head module, and calculate the loss of the output result and the corresponding label data; the parameter update module is used to obtain the comprehensive loss of the task type based on the multiple losses corresponding to multiple data of a single task type, and perform a back propagation based on the comprehensive loss to update the parameters of the shared network module and the task head module; the judgment module is used to judge whether the training end conditions are met; if not, return to obtain multiple data; if so, freeze the parameters of the shared network module.
[0121] Among them, the functional implementation of each module in the above-mentioned multi-task model training device corresponds to the various steps in the above-mentioned multi-task model training method embodiment, and its functions and implementation processes will not be repeated here one by one.
[0122] In a third aspect, the present invention also provides a readable storage medium, on which a multi-task model training program is stored, wherein when the multi-task model training program is executed by a processor, the steps of the above-mentioned multi-task model training method are implemented.
[0123] Among them, the method implemented when the multi-task model training program is executed can refer to the various embodiments of the multi-task model training method of the present invention, and will not be repeated here.
[0124] like Figure 4 and Figure 6 As shown, in a fourth aspect, the present invention further provides a multi-task labeling method, comprising the following steps:
[0125] Input the data to be labeled; process the data using the multi-task model trained by any of the multi-task model training methods described above; and output the labeling results corresponding to each task head module.
[0126] In this example, the data to be labeled is input into the multi-task model trained by any of the multi-task model training methods described above for processing, that is, it is first processed by the shared network module, and then enters the corresponding task head module for processing to obtain the labeling result corresponding to the corresponding task head module.
[0127] In a fifth aspect, the present invention also provides a multi-task labeling device comprising: an input module, a processing module and an output module, wherein the input module is used to input data to be labeled; the processing module is used to process the data using a multi-task model trained by the multi-task model training method described in any of the above items; the output module is used to output the labeling results corresponding to each task head module, such as the classified results (labels), the coordinates or size of the target, etc.
[0128] Among them, the method for training the multi-task model corresponds to the steps in the above-mentioned multi-task model training method embodiment, and its functions and implementation process will not be repeated here one by one.
[0129] In a sixth aspect, a multi-task labeling program is stored on the readable storage medium, wherein when the multi-task labeling program is executed by the processor, the steps of the multi-task labeling method described above are implemented.
[0130] The method implemented when the multi-task labeling program is executed can refer to the various embodiments of the multi-task labeling method of the present invention, and will not be described in detail here.
[0131] Through the description of the above implementation methods, technical personnel in this field can clearly understand that the present application can be implemented through hardware, and can also be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a cooling device, or a network device, etc.) to execute the methods of various implementation scenarios of the present application.
[0132] In summary, by "one-time" backpropagation of the comprehensive loss value corresponding to a single task head module and updating the parameters of the shared network module and the task head module, for multi-task models with poor correlation, the excessive impact of the loss value caused by a single data on the parameters of the shared network module can be reduced, thereby reducing the seesaw effect. In this scheme, the shared network module is updated each time the backpropagation is performed, so this method can reduce the impact of the "seesaw effect" on all task types.
[0133] Similarly, after freezing the parameters of the shared network module, the comprehensive loss value corresponding to the single task head module is back-propagated once and the parameters of the task head module are updated to reduce the impact of the "seesaw effect" on the task head module caused by accidental deviations in the training data. Freezing the parameters of the shared network model and only training and updating the parameters of each task head model can improve training efficiency.
[0134] When the multi-task model training method or device is used to process data, since the method can reduce the quiet board effect, the data processing results of each task type can have a good accuracy rate.
[0135] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0136] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present invention.
[0138] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A multi-task model training method, characterized in that: The following steps are involved: Get multiple data; For each data, the output result is obtained after being processed by the shared network module with fixed parameters and the corresponding task head module, and the loss is calculated between the output result and the corresponding label data; Based on multiple losses corresponding to multiple data of a single task type, the comprehensive loss of the task type is obtained, and a back propagation is performed based on the comprehensive loss to update the parameters of the shared network module and the task head module; Determine whether the training end conditions are met; If not, return to get multiple data; if yes, freeze the parameters of the shared network module.
2. The multi-task model training method according to claim 1, characterized in that: When acquiring multiple data, acquire data of all task types at once, and acquire at least two data of each task type; Each data is processed by a shared network module with fixed parameters and a corresponding task head module, and the loss is calculated; The comprehensive loss of each task type is back-propagated in turn to update the parameters of the shared network module and the task head module.
3. The multi-task model training method according to claim 2, characterized in that: The data of all task types are obtained at one time. When calculating the loss of the data, the data of each task type is processed in turn in a loop, and the number of loops is set based on the amount of data of all task types.
4. The multi-task model training method according to claim 2, characterized in that: When acquiring multiple data, acquire the data of one task type each time; For each data of this task type, the shared network module and the corresponding task header module are processed to obtain the output result, and the loss is calculated; Back propagation is performed based on the comprehensive loss of this task type to update the parameters of the shared network module and the task head module.
5. The multi-task model training method according to claim 4, characterized in that: Obtain the accuracy of the training model corresponding to a sampling of all task type data and the accuracy of the training model corresponding to a sampling of one task type data each time, and select the training model with the highest accuracy as the final training model.
6. The multi-task model training method according to claim 1, characterized in that: After the step of freezing the parameters of the shared network module, the method further includes: Get multiple data; For each data, the output result is obtained after being processed by the shared network module with frozen parameters and the task head module with fixed corresponding parameters, and the loss is calculated for the output result and the corresponding label data; Based on multiple losses corresponding to multiple data of a single task type, the comprehensive loss of the task type is obtained, and a back propagation is performed based on the comprehensive loss to update the parameters of the corresponding task head module; Determine whether the training end conditions are met; If not, return to get more data; if so, end the training.
7. The multi-task model training method according to claim 1, characterized in that: The vector sum of multiple losses corresponding to multiple data of a single task type is taken as the comprehensive loss of this task type.
8. The multi-task model training method according to claim 1, characterized in that: When obtaining multiple data, obtain the data with the corresponding weight ratio according to the weight of each task type.
9. The multi-task model training method according to claim 1, characterized in that: Before obtaining multiple data corresponding to each task type, a static graph of the multi-task flow is pre-generated. The static graph includes the storage addresses of the shared network module and each task header module, as well as the order of data processing for each task type. When executing each task, the shared network module and the corresponding task header module are called based on the disk address in the static graph.
10. A multi-task model training device, characterized in that: include: A data acquisition module, which is used to acquire multiple data; The calculation module is used to obtain the output result after processing each data through the shared network module with fixed parameters and the corresponding task head module, and calculate the loss of the output result and the corresponding label data; A parameter updating module, which is used to obtain a comprehensive loss of a single task type based on multiple losses corresponding to multiple data of the task type, and to perform a back propagation based on the comprehensive loss to update the parameters of the shared network module and the task head module; The judgment module is used to judge whether the training end condition is met; if not, it returns to obtain multiple data; if so, it freezes the parameters of the shared network module.
11. A readable storage medium, characterized in that: The readable storage medium stores a multi-task model training program, wherein when the multi-task model training program is executed by a processor, the steps of the multi-task model training method as described in any one of claims 1 to 9 are implemented.
12. A multi-task labeling method, characterized in that: The following steps are involved: Input the data to be annotated; Processing data using a multi-task model trained by the multi-task model training method according to any one of claims 1 to 9; Output the labeling results corresponding to each task head module.
13. A multi-task labeling device, characterized in that: include: An input module, which is used to input data to be annotated; A processing module, which is used to process data using a multi-task model trained by the multi-task model training method according to any one of claims 1 to 9; The output module is used to output the labeling results corresponding to each task head module.
14. A readable storage medium, characterized in that: The readable storage medium stores a multi-task labeling program, wherein when the multi-task labeling program is executed by a processor, the steps of the multi-task labeling method according to claim 11 are implemented.
Citation Information
Patent Citations
Model training method and device, pedestrian analysis method and device, equipment and storage medium
CN113128478A
Model training method, data processing method, device, equipment, medium and program product
CN118230095A
Systems and methods for deep multi-task learning for embedded machine vision applications
US11527074B1
