Multi-task small-granularity weight averaging method and system based on momentum updating
By using a momentum-updated multi-task small-granularity weight averaging method, the problem of insensitivity to loss changes during training of multi-task learning models is solved, achieving shorter update cycles and smaller effective granularity, thereby improving model stability and training performance.
Patent Information
- Application Number
- CN202410991720.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
Existing multi-task learning models lack sensitivity to batch-level loss changes during training, making stable convergence difficult. Furthermore, existing methods struggle to converge effectively when the loss curve oscillates.
A momentum-based multi-task small-granularity weight averaging method is adopted. By calculating the loss value and ratio of each task and updating the weights in combination with the momentum strategy, the parameters of the multi-task learning model are optimized until the final loss value converges.
It achieves stable convergence of the multi-task learning model during training, responds to short-term weight changes, and comprehensively considers short-term and long-term loss changes, thereby improving the overall training effect.
Smart Images

Figure CN118941919A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and machine learning, and in particular to a multi-task small-granularity weight averaging method and system based on momentum update. Background Art
[0002] Multi-Task Learning (MTL) aims to use the useful information contained in multiple learning tasks to help learn more accurate learners for each task. It is one of the classic applications in the field of machine learning. In the field of traditional machine learning algorithms, multi-task scenarios need to be divided into independent single tasks, and independent hyperparameters and feature extraction methods are customized for each task, resulting in the inability to share information between different tasks. Especially in some related tasks, the information of different tasks can often bring gains to each other, and the independent task settings make them irrelevant and unable to support each other. The advantage of multi-task learning is that different tasks share underlying information during the calculation process. During the training process of related tasks, the optimization processes of different tasks can be linked to each other, which reduces the difficulty of training to a certain extent, while reducing the number of parameters, saving computing resources, and avoiding the risk of overfitting.
[0003] For multi-task learning, how to balance the losses of each task to avoid only achieving effective convergence of one task during the learning process is an extremely important issue. Generally speaking, before training a multi-task model, all losses are scaled to the same order of magnitude with fixed weights and summed to perform multi-task learning. However, in some cases, the convergence speed of a single task is faster, and its corresponding loss exceeds the original order of magnitude, causing the neural network to shift the optimization focus to other tasks, thereby destroying the stable state of the converged task loss. Therefore, how to find a method that can make all tasks reach a stable state of convergence at the same time is the focus of research in the field of multi-task learning.
[0004] At present, Alex Kendall et al. proposed a method to weight multi-task losses based on uncertainty estimation [Kendall A, Gal Y, Cipolla R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7482-7491.], which predicts the cognitive uncertainty and accidental uncertainty of the task through the model, so that the task with smaller uncertainty gets a larger weight, thereby improving the effect of multi-task learning. Michelle Guo et al. proposed a dynamic task priority method [Guo M, Haque A, Huang DA, et al. Dynamic task prioritization for multitask learning [C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 270-287.], using evaluation indicators dedicated to each task, quantifying the optimization degree of the task into a priority value through the evaluation indicator, so that the task branch with poor effect has a larger loss share. Shikun Liu et al. proposed a dynamic weighted balance method to calculate the multi-task loss weights, [Liu S, Johns E, Davison A J. End-to-end multi-task learning with attention [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 1871-1880.], which aims to dynamically calculate the loss weights according to the rate of decline of each loss, using the loss ratio of the current round to the previous round to characterize the rate of decline of the loss, appropriately lowering the weights of tasks with faster loss declines, and amplifying the weights of tasks with slower or even rising loss declines, so as to achieve a balance among multiple tasks. However, since only the historical average loss of the first two rounds is used to update at the initial moment of each round, and the same weight is used for the entire round afterwards, within the round, there is a lack of sensitivity to the loss change state of the batch granularity, resulting in a long period of dynamic change of the task loss weights, which makes the loss curve oscillation difficult to converge effectively.At the same time, the average loss value of historical rounds guides the current round, resulting in a long time span, which makes it difficult to ensure the effectiveness of the guidance. Therefore, it is urgent to redesign a new multi-task loss function to ensure that the multi-task learning model converges stably during the training process. Summary of the invention
[0005] In order to solve the technical problem that the loss change of batch granularity of the existing multi-task learning model lacks sensitivity and cannot converge stably, the embodiment of the present invention provides a multi-task small-granularity weight averaging method and system based on momentum update. The technical solution is as follows:
[0006] On the one hand, a multi-task small-granularity weight averaging method based on momentum update is provided, characterized in that the method includes:
[0007] S1. Based on the original multi-task learning model, obtain the predicted output of each training batch t in each task k and the true value corresponding to the predicted output, and calculate the loss value based on the predicted output and the true value ;
[0008] S2. Starting from the third batch, in each training batch t, the loss values of the first two batches are calculated separately for each task k and Ratio ;
[0009] S3. The ratio of each task k Normalize to get the task weight corresponding to each task k in the current batch t ;
[0010] S4. For each task k, use the momentum strategy combined with the task weight of the current batch t and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ;
[0011] S5: The loss weight of each task k in the current batch t The loss value of the task Multiply and sum to obtain a final loss value, and optimize the parameters of the pre-built multi-task learning model according to the final loss value; repeat S4-S5 for multiple batches until the final loss value of the multi-task learning model converges, and obtain a new multi-task learning model that can be finally applied.
[0012] Optionally, in S1, the loss value calculation method includes: one or more of L1 norm and L2 norm.
[0013] Optionally, in S2, starting from the third batch, the loss values of the first two batches are calculated separately for each task k in each training batch t. and Ratio ,include:
[0014] Starting from the third batch, the loss values of the first two batches are calculated separately for each task k in each training batch t according to the following formula (1): and Ratio :
[0015] (1).
[0016] Optionally, S3, the ratio of each task k Normalize to get the task weight corresponding to each task k in the current batch t ,include:
[0017] According to the following formula (2), the ratio of each task k is Normalize to get the task weight corresponding to each task k in the current batch t :
[0018] (2)
[0019] Among them, T and K are hyperparameters. The size of T represents the similarity of multiple tasks. When T is large, the weights of each task will tend to be the same; K is used to ensure that the sum of the weights is unified.
[0020] Optionally, in S4, for each task k, use the momentum strategy combined with the task weight of the current batch t and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ,include:
[0021] For each task k, according to the following formula (3), the momentum strategy is used in combination with the task weight of the current batch t and the loss weight of the previous batch Calculate the loss weight of the current batch t :
[0022] (3)
[0023] Among them, m is the momentum weight, which ranges from 0 to 1.
[0024] Optionally, in S5, the loss weight of each task k in the current batch t is The loss value of the task Multiply and sum to get the final loss value, including:
[0025] According to the following formula (4), the loss weight of each task k in the current batch t is The loss value of the task Multiply and sum to get the final loss value:
[0026] (4)
[0027] Among them, Loss is the final loss value of the multi-task learning model.
[0028] On the other hand, a multi-task small-granularity weight averaging system based on momentum update is provided, which is applied to a multi-task small-granularity weight averaging method based on momentum update, and the system includes:
[0029] A loss generation module, used to obtain the predicted output of each training batch t in each task k and the true value corresponding to the predicted output based on the original multi-task learning model, and calculate the loss value based on the predicted output and the true value;
[0030] A ratio calculation module is used to calculate the loss values of the first two batches for each task k in each training batch t starting from the third batch. and Ratio ;
[0031] Iterative calculation module, used to convert the ratio of each task k Normalize to get the task weight corresponding to each task k in the current batch t ; For each task k, use the momentum strategy combined with the task weight of the current batch t and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ;
[0032] Model optimization module, used to adjust the loss weight of each task k in the current batch t The loss value of the task Multiply and sum to obtain a final loss value, and optimize the parameters of the pre-built multi-task learning model according to the final loss value; repeat S4-S5 for multiple batches until the final loss value of the multi-task learning model converges, and obtain a new multi-task learning model that can be finally applied.
[0033] On the other hand, a multi-task small-grained weight averaging device based on momentum update is provided, and the multi-task small-grained weight averaging device based on momentum update includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the multi-task small-grained weight averaging methods based on momentum update is implemented.
[0034] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned multi-task small-grained weight averaging methods based on momentum update.
[0035] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0036] The embodiments of the present invention have the following advantages: (1) ease of use: the multi-task small-grained weight averaging method based on momentum update proposed in this embodiment has no hyperparameters and does not require manual adjustment of parameters according to the task type, and can be easily transplanted to different data;
[0037] (2) Excellence: The multi-task small-granularity weight averaging method based on momentum update proposed in this embodiment can guide the multi-task learning model to have a shorter update cycle and a smaller effective granularity during the learning process. The weights can respond to short-term changes and comprehensively consider short-term and long-term loss changes. It can obtain weights that are relatively more favorable for the next loss calculation, thereby achieving overall stability. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0039] Figure 1 It is a flow chart of a multi-task small-granularity weight averaging method based on momentum update provided by an embodiment of the present invention;
[0040] Figure 2 The CT raw data of the cervix, the real synthetic data of PETCT, and the corresponding cancerous area annotation data provided by the embodiment of the present invention;
[0041] Figure 3 Evaluation curves of GAN YOLO and different loss weights provided by the embodiment of the present invention;
[0042] Figure 4Evaluation curves of MT-DBCN and different loss weights provided by the embodiment of the present invention;
[0043] Figure 5 It is a block diagram of a multi-task small-granularity weighted averaging system based on momentum update provided by an embodiment of the present invention;
[0044] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The technical solution of the present invention is described below in conjunction with the accompanying drawings.
[0046] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0047] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.
[0048] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.
[0049] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0050] The embodiment of the present invention provides a multi-task small-granularity weight averaging method based on momentum update, which can be implemented by a multi-task small-granularity weight averaging device based on momentum update, and the multi-task small-granularity weight averaging device based on momentum update can be a terminal or a server. Figure 1 The flowchart of the multi-task small-grained weight averaging method based on momentum update is shown in the figure. The processing flow of the method may include the following steps:
[0051] S101, based on the original multi-task learning model, obtain the predicted output of each training batch t in each task k and the true value corresponding to the predicted output, and calculate the loss value based on the predicted output and the true value ;
[0052] In a feasible implementation, in S101, the loss value calculation method includes: one or more of the L1 norm and the L2 norm.
[0053] S102, starting from the third batch, in each training batch t, the loss values of the first two batches are calculated separately for each task k and Ratio ;
[0054] In one feasible implementation, in S102, starting from the third batch, the loss values of the first two batches are calculated separately for each task k in each training batch t. and Ratio ,include:
[0055] Starting from the third batch, the loss values of the first two batches are calculated separately for each task k in each training batch t according to the following formula (1): and Ratio :
[0056] (1).
[0057] In a feasible implementation, the value range of the training batch t depends on the amount of data in the training set and the setting of the hyperparameter batch size batchsize during model training. For example, if the training set has 120 images and the batch size is set to 4, the maximum batch size is 120 / 4=30.
[0058] S103, the ratio of each task k Normalize to get the task weight corresponding to each task k in the current batch t ;
[0059] In a feasible implementation, S103, the ratio of each task k is Normalize to get the task weight corresponding to each task k in the current batch t ,include:
[0060] According to the following formula (2), the ratio of each task k is Normalize to get the task weight corresponding to each task k in the current batch t :
[0061] (2)
[0062] Among them, T and K are hyperparameters. The size of T represents the similarity of multiple tasks. When T is large, the weights of each task will tend to be the same. K is used to ensure that the sum of the weights is unified.
[0063] S104. For each task k, use the momentum strategy combined with the task weight of the current batch t and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ;
[0064] In one feasible implementation, in S104, for each task k, a momentum strategy is used in combination with the task weight of the current batch t. and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ,include:
[0065] For each task k, according to the following formula (3), the momentum strategy is used in combination with the task weight of the current batch t and the loss weight of the previous batch Calculate the loss weight of the current batch t :
[0066] (3)
[0067] Among them, m is the momentum weight, which ranges from 0 to 1.
[0068] S105. The loss weight of each task k in the current batch t The loss value of the task Multiply and sum to obtain the final loss value, and optimize the parameters of the multi-task learning model based on the final loss value; repeat S4-S5 for multiple batches until the final loss value of the multi-task learning model converges, and obtain a new multi-task learning model that can be finally applied.
[0069] Optionally, in S105, the loss weight of each task k in the current batch t is The loss value of the task Multiply and sum to get the final loss value, including:
[0070] According to the following formula (4), the loss weight of each task k in the current batch t is The loss value of the task Multiply and sum to get the final loss value:
[0071] (4)
[0072] Among them, Loss is the final loss value of the multi-task learning model.
[0073] In a feasible implementation, the method designed by the present invention has a shorter update cycle and a smaller effective granularity, and the weights can respond to short-term changes. It comprehensively considers short-term and long-term loss changes and can obtain weights that are relatively more favorable for the next loss calculation, thereby achieving overall stability.
[0074] In a feasible implementation, the multi-task small-grained weighted averaging method based on momentum update described in this application is described using cervical CT raw data, PETCT real synthetic data, and corresponding cancerous region annotated data. Figure 2 As shown, Figure 2 (a), (b), and (c) are CT annotation images, PET annotation images, and PET / CT and synthesized annotation images, respectively. In this embodiment, the bounding box is indicated by the annotation image of the target detection task.
[0075] In order to fully demonstrate the effectiveness of the momentum-based fine-grained dynamic weight averaging method (MFDWA) proposed in this paper, it is compared with the fixed weighting method and the dynamic weight average method (DWA), and the comparison effect under different momentum weights is adjusted. The GAN YOLO joint method and the MT-DBCN multi-task method are used on cervical lesion images in combination with different multi-task loss weight calculation methods for experimental comparison.
[0076] In order to fully compare the performance of the algorithms, this embodiment uses a 5-fold cross-validation method to divide the data set, that is, the training set, validation set, and test set are divided in a ratio of 8:1:1. During the training process, the model parameters with the best performance on the validation set are saved, and the evaluation results are obtained by testing on the test set.
[0077] Experimental parameters: This experiment includes separate target detection experiments on CT images and PET / CT images, experiments based on the joint method of cross-modal image synthesis and detection, and experiments based on other generation methods and target detection in two stages. Among them, the image generation method is used to convert CT into PETCT based on the joint method of cross-modal image synthesis and detection, and tumors are detected based on PETCT, so this task is a multi-task learning task. For the experiment based on the joint method of cross-modal image synthesis and detection, only the generation part is trained in the first stage. The learning rate is set to 1e-4, the training round is set to 200 rounds, the momentum model is used to update the model parameters and the momentum parameter is set to 0.98, and the optimizer uses the Adam optimizer with a decay rate of 0.001. In the second stage, the training of the target detection part uses the same settings as the above-mentioned target detection method when it is trained separately. In the third stage, the generation and target detection parts are trained at the same time, and each part uses the same settings as described above. For all individual object detection method training, the learning rate is set to 1e-4, the training round is set to 50 rounds, the momentum model with a momentum parameter of 0.98 is used to update the model parameters, and the optimizer is set to a stochastic gradient descent optimizer with a decay rate of 0.01.
[0078] For the other generation methods and two-stage experiments on target detection, the first-stage generation network training and the second-stage target detection training were performed separately, and the training settings of each stage were the same as described above.
[0079] Evaluation indicators: This embodiment selects mAP0.5 and mAP0.5-0.95 as evaluation indicators. The calculation formula is as follows, in which the accuracy and regression rate need to be calculated.
[0080] Precision refers to the proportion of samples classified as positive and correctly to the total number of samples classified as positive. It only focuses on positive samples. The specific calculation method is shown in the following formula (5):
[0081] (5)
[0082] In the formula, TP stands for True Positive, which indicates the number of samples correctly classified as positive samples, and FP stands for False Positive, which indicates the number of samples incorrectly classified as positive samples.
[0083] The recall rate indicates the ratio of the number of samples correctly classified as positive to the total number of positive samples. The specific calculation method is shown in the following formula (6):
[0084] (6)
[0085] In the formula, FN stands for False Negative, which indicates the number of samples that are misclassified as negative samples.
[0086] The way to judge whether a sample is positive or negative is different from that of the classification task. For the target detection task, it is necessary to combine the IOU threshold to judge whether an object is correctly identified. Generally speaking, the IOU threshold is set to 0.5.
[0087] For each category, first match the predicted box with the true box according to the IOU value between them and the pre-specified IOU threshold. Then, calculate the precision and recall of each predicted box based on the matching results. Sort all predicted boxes from high to low according to the confidence level, and then calculate the corresponding precision and recall values according to different confidence thresholds, and draw the Precision-Recall curve. Finally, calculate the area of the curve to get mAP0.5.
[0088] mAP0.5-0.95 is a stricter evaluation index. It needs to calculate the mAP under various threshold conditions within the IoU threshold range of 50-95%, with an increase step of 5%, and then take the average. mAP0.5-0.95 can better evaluate the performance of the model under different IoU thresholds than mAP0.5.
[0089] Experimental results: The experimental results are shown in Tables 1 and 2. It can be seen from Tables 1 and 2 that the performance of the method proposed in this embodiment is better than that of the MFDWA method after setting the momentum update weight m on two multi-task learning tasks. The momentum update makes the task weight change more stable. Among them, when the momentum weight m is 0.5, the MFDWA method has the best overall performance, and the detection accuracy is higher than that of the fixed weight method.
[0090] Table 1 Comparison of different loss weight calculation methods on MT-DBCN
[0091]
[0092] Table 2 Comparison of different loss weight calculation methods on GAN YOLO
[0093]
[0094] like Figure 3 , Figure 4 The evaluation curves of GAN YOLO and MT-DBCN with different loss weights are shown. The DWA method has periodic oscillations during training. In addition, when the momentum weight is 0, the loss weight will respond too much to short-term changes, which is not conducive to long-term stability and leads to the problem of loss explosion at the end of training. When the momentum weight m is 0.5, the MFDWA method performs best overall.
[0095] In the embodiment of the present invention, the multi-task small-granularity weight averaging method based on momentum update has no hyperparameters, does not need to manually adjust parameters according to the task type, and can be easily transplanted to different data; the multi-task small-granularity weight averaging method based on momentum update proposed in this embodiment can guide the multi-task learning model to have a shorter update cycle and a smaller effective granularity during the learning process. The weights can respond to short-term changes, and comprehensively consider short-term and long-term loss changes. It can obtain weights that are relatively more favorable for the next loss calculation, thereby achieving overall stability.
[0096] Figure 5 1 is a block diagram of a multi-task small-granularity weight averaging system based on momentum update according to an exemplary embodiment. The system is used for a multi-task small-granularity weight averaging method based on momentum update. Figure 5 The system includes a loss generation module 310, a ratio calculation module 320, an iterative calculation module 330 and a model optimization module 340. Among them:
[0097] A loss generation module 310 is used to obtain the predicted output of each training batch t in each task k and the true value corresponding to the predicted output based on the original multi-task learning model, and calculate the loss value based on the predicted output and the true value;
[0098] The ratio calculation module 320 is used to calculate the loss values of the first two batches for each task k in each training batch t starting from the third batch. and Ratio ;
[0099] Iterative calculation module 330, used to convert the ratio of each task k Normalize to get the task weight corresponding to each task k in the current batch t ; For each task k, use the momentum strategy combined with the task weight of the current batch t and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ;
[0100] Model optimization module 340, used to adjust the loss weight of each task k in the current batch t The loss value of the task Multiply and sum to obtain a final loss value, and optimize the parameters of the pre-built multi-task learning model according to the final loss value; repeat S4-S5 for multiple batches until the final loss value of the multi-task learning model converges, and obtain a new multi-task learning model that can be finally applied.
[0101] Optionally, the loss value calculation method includes: one or more of L1 norm and L2 norm.
[0102] Optionally, the ratio calculation module 320 is used to calculate the loss values of the first two batches for each task k in each training batch t according to the following formula (1) starting from the third batch: and Ratio :
[0103] (1).
[0104] Optionally, the iterative calculation module 330 is used to convert the ratio of each task k into Normalize to get the task weight corresponding to each task k in the current batch t :
[0105] (2)
[0106] Among them, T and K are hyperparameters. The size of T represents the similarity of multiple tasks. When T is large, the weights of each task will tend to be the same; K is used to ensure that the sum of the weights is unified.
[0107] Optionally, the iterative calculation module 330 is used to use the momentum strategy combined with the task weight of the current batch t for each task k according to the following formula (3): and the loss weight of the previous batch Calculate the loss weight of the current batch t :
[0108] (3)
[0109] Among them, m is the momentum weight, which ranges from 0 to 1.
[0110] Optionally, the model optimization module 340 is used to calculate the loss weight of each task k in the current batch t according to the following formula (4): The loss value of the task Multiply and sum to get the final loss value:
[0111] (4)
[0112] Among them, Loss is the final loss value of the multi-task learning model.
[0113] In the embodiment of the present invention, the multi-task small-granularity weight averaging method based on momentum update has no hyperparameters, does not need to manually adjust parameters according to the task type, and can be easily transplanted to different data; the multi-task small-granularity weight averaging method based on momentum update proposed in this embodiment can guide the multi-task learning model to have a shorter update cycle and a smaller effective granularity during the learning process. The weights can respond to short-term changes, and comprehensively consider short-term and long-term loss changes. It can obtain weights that are relatively more favorable for the next loss calculation, thereby achieving overall stability.
[0114] Figure 6 is a structural diagram of a multi-task small-granularity weight averaging device based on momentum update provided by an embodiment of the present invention, such as Figure 6 As shown, the multi-task small-grained weight averaging device based on momentum update may include the above Figure 5 The multi-task small-granularity weight averaging system based on momentum update is shown. Optionally, the multi-task small-granularity weight averaging device 410 based on momentum update may include a first processor 2001 .
[0115] Optionally, the momentum-updating based multi-task small-granularity weight averaging device 410 may further include a memory 2002 and a transceiver 2003 .
[0116] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.
[0117] Combine the following Figure 6 The components of the multi-task small-granularity weight averaging device 410 based on momentum update are specifically introduced as follows:
[0118] The first processor 2001 is the control center of the multi-task small-granularity weight averaging device 410 based on momentum update, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).
[0119] Optionally, the first processor 2001 may perform various functions of the momentum-updated multi-task small-granularity weight averaging device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .
[0120] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 6 CPU0 and CPU1 are shown in FIG.
[0121] In a specific implementation, as an embodiment, the multi-task small-granularity weight averaging device 410 based on momentum update may also include multiple processors, such as Figure 6 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0122] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.
[0123] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0124] The transceiver 2003 is used to communicate with a network device or a terminal device.
[0125] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 6 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.
[0126] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0127] It should be noted that Figure 6 The structure of the momentum-updating-based multi-task small-granularity weight averaging device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0128] In addition, the technical effects of the multi-task small-granularity weight averaging device 410 based on momentum update can refer to the technical effects of the multi-task small-granularity weight averaging method based on momentum update described in the above method embodiment, which will not be repeated here.
[0129] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0130] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0131] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0132] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.
[0133] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0134] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0136] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.
[0138] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A multi-task small-granularity weight averaging method based on momentum update, characterized in that: The method comprises: S1. Based on the original multi-task learning model, obtain the predicted output of each training batch t in each task k and the true value corresponding to the predicted output, and calculate the loss value based on the predicted output and the true value ; S2. Starting from the third batch, in each training batch t, the loss values of the first two batches are calculated separately for each task k and Ratio ; S3. The ratio of each task k Normalize to get the task weight corresponding to each task k in the current batch t ; S4. For each task k, use the momentum strategy combined with the task weight of the current batch t and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ; S5: The loss weight of each task k in the current batch t The loss value of the task Multiply and sum to obtain a final loss value, and optimize the parameters of the pre-built multi-task learning model according to the final loss value; repeat S4-S5 for multiple batches until the final loss value of the multi-task learning model converges, and obtain a new multi-task learning model that can be finally applied.
2. The multi-task small-granularity weight averaging method based on momentum update according to claim 1 is characterized in that: In S1, the loss value calculation method includes: one or more of the L1 norm and the L2 norm.
3. The multi-task small-granularity weight averaging method based on momentum update according to claim 2 is characterized in that: In S2, starting from the third batch, the loss values of the first two batches are calculated separately for each task k in each training batch t. and Ratio ,include: Starting from the third batch, the loss values of the first two batches are calculated separately for each task k in each training batch t according to the following formula (1): and Ratio : (1)。 4. The multi-task small-granularity weight averaging method based on momentum update according to claim 3 is characterized in that: S3, the ratio of each task k Normalize to get the task weight corresponding to each task k in the current batch t ,include: According to the following formula (2), the ratio of each task k is Normalize to get the task weight corresponding to each task k in the current batch t : (2) Among them, T and K are hyperparameters. The size of T represents the similarity of multiple tasks. When T is large, the weights of each task will tend to be the same; K is used to ensure that the sum of the weights is unified.
5. The multi-task small-granularity weight averaging method based on momentum update according to claim 4 is characterized in that: In S4, for each task k, the momentum strategy is used in combination with the task weight of the current batch t. and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ,include: For each task k, according to the following formula (3), the momentum strategy is used in combination with the task weight of the current batch t and the loss weight of the previous batch Calculate the loss weight of the current batch t : (3) Among them, m is the momentum weight, which ranges from 0 to 1.
6. The multi-task small-granularity weight averaging method based on momentum update according to claim 5 is characterized in that: In S5, the loss weight of each task k in the current batch t is The loss value of the task Multiply and sum to get the final loss value, including: According to the following formula (4), the loss weight of each task k in the current batch t is The loss value of the task Multiply and sum to get the final loss value: (4) Among them, Loss is the final loss value of the multi-task learning model.
7. A multi-task small-granularity weight averaging system based on momentum update, the multi-task small-granularity weight averaging system based on momentum update is used to implement the multi-task small-granularity weight averaging method based on momentum update as claimed in any one of claims 1 to 6, characterized in that: The system comprises: A loss generation module, used to obtain the predicted output of each training batch t in each task k and the true value corresponding to the predicted output based on the original multi-task learning model, and calculate the loss value based on the predicted output and the true value; A ratio calculation module is used to calculate the loss values of the first two batches for each task k in each training batch t starting from the third batch. and Ratio ; Iterative calculation module, used to convert the ratio of each task k Normalize to get the task weight corresponding to each task k in the current batch t ; For each task k, use the momentum strategy combined with the task weight of the current batch t and the loss weight of the previous batch Calculate and get the loss weight of the current batch t ; Model optimization module, used to adjust the loss weight of each task k in the current batch t The loss value of the task Multiply and sum to obtain a final loss value, and optimize the parameters of the pre-built multi-task learning model according to the final loss value; repeat S4-S5 for multiple batches until the final loss value of the multi-task learning model converges, and obtain a new multi-task learning model that can be finally applied.
8. The multi-task small-granularity weight averaging system based on momentum update according to claim 7 is characterized in that: The loss value calculation method includes: one or more of L1 norm and L2 norm.
9. A multi-task small-granularity weight averaging device based on momentum update, characterized in that: The multi-task small-granularity weight averaging device based on momentum update includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Alternating training method based on model fusion
CN115035337A
Multi-mode hatred model cause detection method based on multi-task learning network
CN115497105A
Cited By
Multi-task deep optimization method based on dynamic weight distribution
CN120994339A
Multi-task deep optimization method based on dynamic weight distribution
CN120994339B
