Model training method and device

By introducing adaptive adjustment parameters into the model distillation technology, the problem of time-consuming determination of the weight of the distillation loss function is solved, which improves training efficiency and reduces costs, and achieves more efficient model training.

CN118940852APending Publication Date: 2024-11-12BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310517854.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the existing model distillation technology, it takes a long time to determine the optimal solution process of distillation loss function weight, affects training efficiency and increases costs.

Method used

By introducing adaptive adjustment parameters, the weights of the truth loss function and the distillation loss function are adaptively adjusted during the training process, avoiding complex weight adjustment processes.

Benefits of technology

This improves the efficiency of model training, reduces costs, ensures that the distillation loss function plays full role in the training process, and improves the training accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940852A_ABST
    Figure CN118940852A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a model training method and equipment, and the method comprises the steps: obtaining a training sample, inputting the training sample into a student model and a teacher model, obtaining a first output value and a second output value, determining a truth value loss function and a distillation loss function according to the first output value, the second output value and a truth value label, and obtaining a distillation loss function; and determining adaptive adjustment parameters, determining a total loss function according to the truth value loss function, the distillation loss function and the adaptive adjustment parameters, and training a student model according to the total loss function. According to the model training method provided by the embodiment of the invention, the adaptive adjustment parameters are determined and added, so that the weights of the truth value loss function and the distillation loss function can be adaptively adjusted in the training process, the training efficiency is improved, a complex weight adjustment process is avoided, and the cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular, to a model training method and device. Background Art

[0002] Model distillation refers to the use of a higher-precision teacher model (larger parameters) to guide the training of a lower-precision student model (smaller parameters and faster prediction speed) in deep learning model training, thereby improving the accuracy of the small model while ensuring the inference speed.

[0003] In the related art, during the training process, it is necessary to first determine the optimal solution for the weight of the true value loss function by conducting multiple groups of experiments, so as to establish a more reasonable total loss function based on the optimal solution for training.

[0004] However, the inventors have found that there are at least the following technical problems in the related art: the above process of determining the optimal solution for the weights of the distillation loss function is time-consuming, increases costs, and affects the overall training efficiency. Summary of the invention

[0005] The embodiments of the present disclosure provide a model training method and device to reduce costs and improve overall training efficiency.

[0006] In a first aspect, an embodiment of the present disclosure provides a model training method, comprising:

[0007] Obtain training samples, and input the training samples into a student model and a teacher model respectively, obtain a first output value and a second output value, and determine a true value loss function and a distillation loss function according to the first output value, the second output value and the true value label of the training sample;

[0008] Determining adaptive adjustment parameters;

[0009] Determine a total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameter;

[0010] The student model is trained according to the total loss function.

[0011] In a second aspect, the present disclosure provides an image processing method, including:

[0012] Get the image to be processed;

[0013] The image to be processed is input into a deep learning model to obtain a posture estimation result corresponding to the image to be processed; the deep learning model is obtained by training according to the model training method described in the first aspect and various possible designs of the first aspect.

[0014] In a third aspect, an embodiment of the present disclosure provides a model training device, including:

[0015] An input module, used to obtain training samples, and input the training samples into the student model and the teacher model respectively, to obtain a first output value and a second output value, and to determine a true value loss function and a distillation loss function according to the first output value, the second output value and the true value label of the training sample;

[0016] A determination module, used to determine adaptive adjustment parameters;

[0017] The determination module is further used to determine the total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameter;

[0018] A training module, used to train the student model according to the total loss function

[0019] In a fourth aspect, an embodiment of the present disclosure provides an image processing device, including:

[0020] An acquisition module, used for acquiring an image to be processed;

[0021] A detection module is used to input the image to be processed into a deep learning model for posture estimation to obtain a posture estimation result corresponding to the image to be processed; the deep learning model is trained according to the model training method described in the first aspect and various possible designs of the first aspect.

[0022] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0023] The memory stores computer-executable instructions;

[0024] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the model training method described in the first aspect and various possible designs of the first aspect, and the second aspect and various possible designs of the second aspect.

[0025] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the model training method described in the first aspect and various possible designs of the first aspect, and the second aspect and various possible designs of the second aspect are implemented.

[0026] In the seventh aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the model training method described in the first aspect and various possible designs of the first aspect, and the second aspect and various possible designs of the second aspect.

[0027] The model training method and device provided in this embodiment include obtaining training samples, and inputting the training samples into a student model and a teacher model respectively, obtaining a first output value and a second output value, determining a truth loss function and a distillation loss function according to the first output value, the second output value and the truth value label, determining an adaptive adjustment parameter, determining a total loss function according to the truth loss function, the distillation loss function and the adaptive adjustment parameter, and training the student model according to the total loss function. The model training method provided in this embodiment determines and adds adaptive adjustment parameters, so that the weights of the truth loss function and the distillation loss function can be adaptively adjusted during the training process, thereby improving training efficiency, avoiding complex weight adjustment processes, and reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A schematic diagram of the principle of the model training method provided in the embodiment of the present disclosure;

[0030] Figure 2 Schematic diagram of the model training method provided in the embodiment of the present disclosure Figure 1 ;

[0031] Figure 3 Schematic diagram of the model training method provided in the embodiment of the present disclosure Figure 2 ;

[0032] Figure 4 A curve diagram of a monotone non-increasing function provided in an embodiment of the present disclosure Figure 1 ;

[0033] Figure 5 A curve diagram of a monotone non-increasing function provided in an embodiment of the present disclosure Figure 2 ;

[0034] Figure 6 A schematic diagram of the image processing method provided by the embodiment of the present disclosure;

[0035] Figure 7 A structural block diagram of a model training device provided in an embodiment of the present disclosure;

[0036] Figure 8 A structural block diagram of an image processing device provided by an embodiment of the present disclosure;

[0037] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0039] In deep learning tasks, models with large model parameters tend to have high accuracy but slow prediction speed, while models with small model parameters tend to have fast prediction speed but low accuracy. Model distillation refers to using a high-precision teacher model (larger parameters) to guide the training of a low-precision student model (smaller parameters, faster prediction speed), thereby improving the accuracy of the small model while ensuring the inference speed.

[0040] In the process of model distillation, the student model is usually trained by using the total loss function obtained by the weighted sum of the true value loss function (representing the loss value between the student model output and the true value label) and the distillation loss function (representing the loss value between the student model output and the teacher model output). Model distillation was first proposed in tasks such as image classification. Generally, the KL divergence of the output probability distribution is used as the loss function to guide the training process of the student model. Later, the idea of ​​feature distillation was introduced in tasks such as target detection. The feature map distance between the calculated student model and the teacher model is used as the loss function to guide the training process of the student model.

[0041] However, during the feature distillation process, the distillation loss function can easily converge to a value close to 0 quickly. This means that even if the distillation loss function is added during training, the distillation method still cannot work. If the proportion of the distillation loss function is excessively increased during training, the training process will be biased and unable to focus on the learning process of the task itself.

[0042] Moreover, in the related art, during the training process, it is necessary to first determine the optimal solution of the weight of the true value loss function through multiple groups of experiments, so as to establish a more reasonable total loss function based on the optimal solution, so that the model trained based on the total loss function has higher accuracy. Specifically, a weight value is assumed in each experiment, and then a total loss function is constructed based on the assumed weight value for training to obtain the accuracy of the student model trained in this experiment, and finally the weight value corresponding to the highest accuracy is selected as the optimal solution based on the accuracy corresponding to each experiment. For example: in the image classification task, the optimal solution of the weight of the distillation loss function is 1, and in the detection task, the optimal solution of the weight of the distillation loss function is 20. If it is set to 1, the training effect will even be negative. However, the above process of determining the optimal solution of the weight of the distillation loss function is time-consuming, increases costs, and affects the overall training efficiency.

[0043] In order to solve the above technical problems, the inventors of the present application have found that by designing adaptive adjustment parameters, the weights can be adaptively changed during the model training process to improve the training efficiency and avoid the complicated weight adjustment process. Based on this, the present disclosure provides a model training method.

[0044] refer to Figure 1 , Figure 1 The schematic diagram of the principle of the model training method provided in the embodiment of the present disclosure is as follows. Figure 1 As shown, the training sample set includes multiple training samples, each training sample has a true value label. In each iteration round, a training sample is selected from the training sample set and input into the teacher model and the student model. The true value loss function is determined according to the first output value output by the student model and the true value label corresponding to the training sample. The distillation loss function is determined according to the first output value and the second output value output by the teacher model. The adaptive adjustment parameter is determined (exemplarily, the adaptive adjustment parameter can be determined according to the scalar of the true value loss function and the scalar of the distillation loss function, and the adaptive adjustment parameter can also be determined according to the iteration round and the total iteration round, and the above two methods can be combined to determine the adaptive adjustment parameter). Then, according to the true value loss function, the distillation loss function and the adaptive adjustment parameter, the total loss function is determined, wherein the adaptive adjustment parameter is used to adjust the amplitude of the distillation loss function so that the ratio between the amplitude of the true value loss function and the amplitude of the adjusted distillation loss function within the preset training period is within a preset range. Finally, the student model is trained according to the total loss function. In the disclosed embodiment, adaptive adjustment parameters are determined and added so that the weights of the true value loss function and the distillation loss function can be adaptively adjusted during the training process, thereby improving training efficiency, avoiding complex weight adjustment processes, and reducing costs.

[0045] refer to Figure 2 , Figure 2Schematic diagram of the model training method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied to a model training device such as a terminal device or a server. The model training method includes:

[0046] 201. Obtain training samples, and input the training samples into a student model and a teacher model respectively to obtain a first output value and a second output value, and determine a true value loss function and a distillation loss function according to the first output value, the second output value and the true value label of the training sample.

[0047] In this embodiment, the training sample can be an image sample. The teacher model can be a deep learning model with a larger number of parameters and higher precision than the student model. The student model can be a deep learning model with a smaller number of parameters than the teacher model. The teacher model and the learning model can be used for classification tasks, target detection tasks, or optical character recognition (OCR) text recognition tasks. The first output value obtained by inputting the training sample into the student model and the second output value obtained by inputting the training sample into the teacher model can be the category probability distribution in the classification task, the intermediate layer feature plane (Feature Map) in the target detection task, or the predicted text probability distribution in the OCR task.

[0048] The true value label can be the true value that is pre-labeled for the training sample. For example, in a classification task (such as classifying an animal in an image as a cat or a dog), the true value label refers to the category to which the labeled training sample belongs (for example, if the training sample is an image of a cat, it is labeled with the category corresponding to the cat).

[0049] A truth loss function may be determined according to the first output value and the truth label, and a distillation loss function may be determined according to the first output value and the second output value, wherein the truth loss function represents the loss between the student model output and the truth value, and the distillation loss function represents the loss between the teacher model output and the student model output.

[0050] 202. Determine adaptive adjustment parameters.

[0051] Specifically, there are multiple ways to determine the adaptive adjustment parameters.

[0052] In one implementable manner, an adaptive adjustment parameter can be determined based on the scalar of the true value loss function and the scalar of the distillation loss function; the adaptive adjustment parameter is used to adjust the amplitude of the distillation loss function so that the ratio between the amplitude of the true value loss function and the adjusted amplitude of the distillation loss function within a preset training period is within a preset range.

[0053] Among them, the true value loss function and the distillation loss function are both vectors, and their scalar refers to the amount after removing the gradient, that is, the amplitude.

[0054] Specifically, the inventors have discovered through research that one of the reasons for the change in the optimal solution of weights in different tasks is that the difference in the speed of change of the amplitude of the distillation loss function and the amplitude of the true value loss function is too large. Therefore, in the process of model distillation, an adaptive distillation loss function adjustment strategy related to the scalars of the distillation loss function and the true value loss function can be introduced. This not only avoids selecting the optimal solution of weights through a complex parameter adjustment process and reduces costs, but also realizes dynamic parameter adjustment through the adaptive distillation loss function, which enables the distillation loss function to play a full role in the training process, ensures the beneficial effects of the distillation scheme, and improves training efficiency.

[0055] In some embodiments, determining the adaptive adjustment parameter based on the scalar of the true value loss function and the scalar of the distillation loss function may include: calculating the ratio between the scalar of the true value loss function and the scalar of the distillation loss function based on the scalar of the true value loss function and the scalar of the distillation loss function; determining the adaptive adjustment parameter based on the ratio between the scalar of the true value loss function and the scalar of the distillation loss function.

[0056] For example, the true value loss function can be L gt , let the distillation loss function be L dist , then the adaptive adjustment parameter can be g(|L gt |, L dist |), specifically, g(|L gt |,|L dist |) can be (but not limited to)

[0057] The method of this embodiment, by adding an adaptive adjustment parameter, is equivalent to assigning an adaptively adjustable weight to the distillation loss function, thereby realizing automatic adjustment of the amplitude of the distillation loss function, ensuring that the amplitude of the distillation loss function is equivalent to the amplitude of the true value loss function within a preset time period during the training process, for example, they are of the same dimension, thereby enabling the distillation loss function to fully play its role, improving training efficiency, and avoiding the need to conduct multiple experiments to obtain the optimal solution through manual adjustment of the weights, thereby saving costs.

[0058] In another possible implementation, the adaptive adjustment parameter may be determined according to the current iteration round corresponding to the training sample.

[0059] Specifically, f(ce, te) can be an adaptive adjustment parameter. f(ce, te) is a function of the current iteration round ce and the total iteration round te, and is a monotonic non-increasing function of ce. te is a constant. For example, the change of f(ce, te) can be a larger value in the early stage of training to make the model converge quickly; it can decrease with the increase of the number of iteration rounds ce in the middle of training; and it can be a smaller value in the late stage of training to ensure that the model can converge stably in the task. Some common f(ce, te) curves can be as follows Figure 4 or Figure 5 As shown. The horizontal axis is the current round ce, and the vertical axis is f(ce, re).

[0060] In another possible implementation, the above two methods can be combined. For example, the adaptive adjustment parameter can be determined according to the scalar of the truth loss function, the scalar of the distillation loss function and the current iteration round corresponding to the training sample. Figure 3 The description of the illustrated embodiment will not be repeated here.

[0061] 203. Determine a total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameter.

[0062] Specifically, the adaptive adjustment parameter can be combined with the true value loss function and the distillation loss function, or one of the two can be combined with the adaptive adjustment parameter to obtain the total loss function. For example, the adaptive adjustment parameter can be multiplied with the true value loss function and then added to the distillation loss function to obtain the total loss function, or the adaptive adjustment parameter can be multiplied with the distillation loss function and then added to the true value loss function to obtain the total loss function.

[0063] In some embodiments, determining a total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameters may include: determining a distillation loss adaptive function according to the distillation loss function and the adaptive adjustment parameters; determining a total loss function according to the true value loss function and the distillation loss adaptive function.

[0064] Optionally, determining the distillation loss adaptive function according to the distillation loss function and the adaptive adjustment parameter may include: calculating the product of the distillation loss function and the adaptive adjustment parameter according to the distillation loss function and the adaptive adjustment parameter; determining the distillation loss adaptive function according to the product of the distillation loss function and the adaptive adjustment parameter.

[0065] Optionally, determining the total loss function according to the true value loss function and the distillation loss adaptive function may include: performing weighted summation of the true value loss function and the distillation loss adaptive function to obtain the total loss function.

[0066] For example, the adaptive adjustment parameter is g(|L gt |,|L dist |) as an example. The total loss function can be expressed as follows:

[0067] L all-new =α*L gt +β*g(|L gt |,|L dist |)*L dist (1)

[0068] Among them, g(|L gt |,|L dist |) is the adaptive adjustment parameter of the distillation loss function, α and β are the weight coefficients of the true value loss function and the distillation loss function, which are generally constants, and L gt is the true value loss function, L dist is the distillation loss function. In this embodiment, an adaptive adjustment parameter g(|L gt |,|L dist |), adjust the distillation loss function so that the ratio between the amplitude of the true value loss function and the amplitude of the adjusted distillation loss function within a preset training period is within a preset range, thereby achieving class normalization of the distillation loss function. This can avoid the need to determine the weight ratio of the distillation loss function through a large number of experiments, and at the same time alleviate the problem that the distillation loss function cannot fully function due to a rapid decrease in the distillation process of target detection tasks such as gesture estimation.

[0069] 204. Train the student model according to the total loss function.

[0070] Specifically, after calculating the total loss function, the internal parameters of the student model can be adjusted based on the total loss function to complete the training of the current iteration round. If the current iteration round is greater than or equal to the total iteration rounds or the accuracy is less than or equal to the preset accuracy, the training can be terminated. If the current iteration round is less than the total iteration rounds and the accuracy is greater than the preset accuracy, the next iteration round can be entered, and steps 201 to 204 can be repeated to obtain the training samples of the next iteration round. After calculating the total loss function of the next iteration round, the student model is trained based on the total loss function of the next iteration round until the training end condition is met (the total iteration rounds are reached, or the preset accuracy is reached).

[0071] From the above description, it can be seen that by determining and adding adaptive adjustment parameters, the weights of the true value loss function and the distillation loss function can be adaptively adjusted during the training process, thereby improving training efficiency, avoiding complex weight adjustment processes, and reducing costs.

[0072] refer to Figure 3 , Figure 3 Schematic diagram of the model training method provided in the embodiment of the present disclosure Figure 2 In order to solve the problem that the distillation loss function affects the convergence of the model in the later stage during the model distillation process, the factor of the iteration round is added to the adaptive adjustment parameter in this embodiment. The model training method includes:

[0073] 301. Obtain training samples, and input the training samples into a student model and a teacher model respectively to obtain a first output value and a second output value, and determine a true value loss function and a distillation loss function according to the first output value, the second output value and the true value label of the training sample.

[0074] Step 301 in this embodiment is similar to step 201 in the above embodiment and will not be described again here.

[0075] 302. Determine an adaptive adjustment parameter according to the scalar of the true value loss function, the scalar of the distillation loss function, and the current iteration round corresponding to the training sample.

[0076] The truth loss function can be determined according to the first output value and the truth label, and the distillation loss function can be determined according to the first output value and the second output value. The truth loss function represents the loss between the student model output and the truth value, and the distillation loss function represents the loss between the teacher model output and the student model output. The truth loss function and the distillation loss function are both vectors, and their scalar refers to the amount after the gradient is removed, that is, the amplitude.

[0077] In some embodiments, determining the adaptive adjustment parameter based on the scalar of the truth loss function, the scalar of the distillation loss function and the current iteration round may include: determining the adaptive adjustment coefficient based on the scalar of the truth loss function and the scalar of the distillation loss function; obtaining the preset total iteration rounds and the current iteration round corresponding to the training sample; determining a monotonic non-increasing function based on the current iteration round and the total iteration rounds; determining the adaptive adjustment parameter based on the adaptive adjustment coefficient and the monotonic non-increasing function.

[0078] Optionally, determining the adaptive adjustment coefficient according to the scalar of the true value loss function and the scalar of the distillation loss function may include: calculating the ratio between the scalar of the true value loss function and the scalar of the distillation loss function according to the scalar of the true value loss function and the scalar of the distillation loss function; determining the adaptive adjustment parameter according to the ratio between the scalar of the true value loss function and the scalar of the distillation loss function.

[0079] For example, the true value loss function can be L gt, let the distillation loss function be L dist , then the adaptive adjustment parameter can be g(|L gt |, L dist |)*f(ce,te), where g(L gt |,|L dist |) is the adaptive adjustment coefficient. Specifically, g(|L gt |,|L dist |) can be (but not limited to) f(ce, te) is a function of the current iteration round ce and the total iteration round te, and is a monotonic non-increasing function of ce. te is a constant. Exemplarily, the change of f(ce, te) can be a larger value in the early stage of training, so that the model converges quickly; it decreases with the increase of the number of iteration rounds ce in the middle of training; in the late stage of training, it is a smaller value to ensure that the model can converge stably in the task. Some common f(ce, te) curves can be shown in Figure 4 or Figure 5. The horizontal axis is the current round ce, and the vertical axis is f(ce, te).

[0080] 303. Determine a total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameter;

[0081] Specifically, adaptive adjustment parameters related to the scalar of the true value loss function, the scalar of the distillation loss function and the current iteration round are introduced to adjust the distillation loss function so that the ratio between the amplitude of the true value loss function and the amplitude of the adjusted distillation loss function within the preset training period is within the preset range, so as to achieve class normalization of the distillation loss function, avoid the process of determining the weight ratio of the distillation loss function through a large number of experiments, and alleviate the problem that the distillation loss function cannot fully play its role due to the rapid decrease in the distillation process of target detection tasks such as gesture estimation. And through the monotonic non-increasing property of f(ce, te), it is ensured that the distillation training focuses on feature imitation and learning in the early stage, and focuses on the convergence process of specific tasks in the later stage, so that the key elements of the training process can be focused on in both the early and late stages of training.

[0082] In some embodiments, determining a total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameters may include: determining a distillation loss adaptive function according to the distillation loss function and the adaptive adjustment parameters; determining a total loss function according to the true value loss function and the distillation loss adaptive function.

[0083] Optionally, determining the distillation loss adaptive function according to the distillation loss function and the adaptive adjustment parameter may include: calculating the product of the distillation loss function and the adaptive adjustment parameter according to the distillation loss function and the adaptive adjustment parameter; determining the distillation loss adaptive function according to the product of the distillation loss function and the adaptive adjustment parameter.

[0084] Optionally, determining the total loss function according to the true value loss function and the distillation loss adaptive function may include: performing weighted summation of the true value loss function and the distillation loss adaptive function to obtain the total loss function.

[0085] Exemplarily, the total loss function can be expressed as follows:

[0086] L all-new =α*L gt +β*g(|Lg t |,|L dist |)*f(ce,te)*L dist (2)

[0087] Among them, g(|L gt ,|L dist |)*f(ce, te) is the adaptive adjustment parameter of the distillation loss function, α and β are the weight coefficients of the true value loss function and the distillation loss function, which are generally constants, L gt is the true value loss function, L dist is the distillation loss function. g(|L gt |,|L dist |) is the adaptive adjustment coefficient. Specifically, g(|L gt |,|L dist |) can be (but not limited to) f(ce, te) is a function of the current iteration ce and the total iteration te, and is a monotonic non-increasing function of ce. te is a constant. For example, the change of f(ce, te) can be a larger value in the early stage of training to make the model converge quickly; it can decrease with the increase of the number of iterations ce in the middle of training; and it can be a smaller value in the late stage of training to ensure that the model can converge stably in the task. Some common f(ce, te) curves can be as follows: Figure 4 or Figure 5 As shown. The horizontal axis is the current round ce, and the vertical axis is f(ce, te).

[0088] 304. Train the student model according to the total loss function.

[0089] Step 304 in this embodiment is similar to step 204 in the above embodiment and will not be described again here.

[0090] From the above description, we can see that by introducing g(|L gt |,|L dist |), classify the distillation loss function, thereby avoiding the need to determine the weight ratio of the distillation loss function through a large number of experiments, and at the same time alleviating the problem of the distillation loss function decreasing during the distillation process of tasks such as gesture estimation. By introducing f(ce, te), the monotonic non-increasing property of this function ensures that the early stage of distillation training focuses on feature imitation and learning, and the later stage of training focuses on the convergence process for specific tasks.

[0091] refer to Figure 6 , Figure 6 A schematic diagram of the image processing method provided by the embodiment of the present disclosure. The execution subject of this embodiment may be a terminal device. The image processing method includes:

[0092] 601. Obtain an image to be processed.

[0093] 602. Input the image to be processed into a deep learning model to obtain a posture estimation result corresponding to the image to be processed. The deep learning model is obtained by training according to the model training method described in the above embodiment.

[0094] Specifically, in the above embodiments, for example Figure 2 or Figure 3 After the model training method described in the illustrated embodiment trains the student model to obtain a deep learning model, the deep model can be set on a terminal device (mobile phone, tablet, computer, etc.), and the image to be processed can be input into the deep learning model to perform classification tasks, target detection tasks or OCR text recognition tasks.

[0095] From the above description, it can be seen that by determining and adding adaptive adjustment parameters, the weights of the true value loss function and the distillation loss function can be adaptively adjusted during the training process, thereby improving training efficiency, avoiding complex weight adjustment processes, and reducing costs.

[0096] Corresponding to the model training method in the above embodiment, Figure 7 This is a structural block diagram of a model training device provided in an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 7 The device 70 includes: an input module 701, a determination module 702 and a training module 703.

[0097] Among them, the input module 701 is used to obtain training samples, and input the training samples into the student model and the teacher model respectively, to obtain the first output value and the second output value, and to determine the true value loss function and the distillation loss function according to the first output value, the second output value and the true value label of the training sample.

[0098] The determination module 702 is used to determine the adaptive adjustment parameters.

[0099] The determination module 702 is further used to determine the total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameter.

[0100] The training module 703 is used to train the student model according to the total loss function.

[0101] In one embodiment of the present disclosure, the determination module 702 is specifically used to: determine an adaptive adjustment parameter based on the scalar of the true value loss function and the scalar of the distillation loss function; the adaptive adjustment parameter is used to adjust the amplitude of the distillation loss function so that the ratio between the amplitude of the true value loss function and the adjusted amplitude of the distillation loss function within a preset training period is within a preset range.

[0102] In one embodiment of the present disclosure, the determination module 702 is specifically used to: calculate the ratio between the scalar of the true value loss function and the scalar of the distillation loss function according to the scalar of the true value loss function and the scalar of the distillation loss function; determine the adaptive adjustment parameter according to the ratio between the scalar of the true value loss function and the scalar of the distillation loss function.

[0103] In one embodiment of the present disclosure, the determination module 702 is specifically configured to: determine the adaptive adjustment parameter according to the current iteration round corresponding to the training sample.

[0104] In one embodiment of the present disclosure, the determination module 702 is specifically used to: obtain a preset total iteration round and a current iteration round corresponding to the training sample; determine a monotonic non-increasing function based on the current iteration round and the total iteration round; and determine an adaptive adjustment parameter based on the monotonic non-increasing function.

[0105] In one embodiment of the present disclosure, the determination module 702 is specifically used to: determine a distillation loss adaptive function according to the distillation loss function and the adaptive adjustment parameter; and determine a total loss function according to the true value loss function and the distillation loss adaptive function.

[0106] In one embodiment of the present disclosure, the determination module 702 is specifically used to: calculate the product of the distillation loss function and the adaptive adjustment parameter according to the distillation loss function and the adaptive adjustment parameter; determine the distillation loss adaptive function according to the product of the distillation loss function and the adaptive adjustment parameter.

[0107] In one embodiment of the present disclosure, the determination module 702 is specifically used to: perform weighted summation of the true value loss function and the distillation loss adaptive function to obtain a total loss function.

[0108] In one embodiment of the present disclosure, the training sample is an image sample; the determination module 702 is specifically used to: calculate the feature map distance between the student model and the teacher model according to the first output value and the second output value, and determine the feature map distance as the distillation loss function.

[0109] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0110] Corresponding to the image processing method of the above embodiment, Figure 8 This is a structural block diagram of an image processing device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8 The device 80 includes: an acquisition module 801 and a detection module 802.

[0111] An acquisition module 801 is used to acquire an image to be processed;

[0112] The detection module 802 is used to input the image to be processed into a deep learning model for posture estimation to obtain a posture estimation result corresponding to the image to be processed; the deep learning model is obtained by training according to the model training method described in the above embodiment.

[0113] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0114] In order to implement the above embodiments, the embodiments of the present disclosure also provide an electronic device for executing the image processing method or model training method provided in the above embodiments.

[0115] refer to Fig. 9, which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0116] like Fig. 9 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0117] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0118] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0119] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0120] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0121] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0122] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0125] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0126] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0128] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0129] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A model training method, characterized in that: include: Obtain training samples, and input the training samples into a student model and a teacher model respectively, obtain a first output value and a second output value, and determine a true value loss function and a distillation loss function according to the first output value, the second output value and the true value label of the training sample; Determining adaptive adjustment parameters; Determine a total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameter; The student model is trained according to the total loss function.

2. The method according to claim 1, characterized in that The step of determining the adaptive adjustment parameter comprises: An adaptive adjustment parameter is determined according to the scalar of the true value loss function and the scalar of the distillation loss function; the adaptive adjustment parameter is used to adjust the amplitude of the distillation loss function so that the ratio between the amplitude of the true value loss function and the adjusted amplitude of the distillation loss function within a preset training period is within a preset range.

3. The method according to claim 2, characterized in that The step of determining the adaptive adjustment parameter according to the scalar of the truth loss function and the scalar of the distillation loss function includes: According to the scalar of the true value loss function and the scalar of the distillation loss function, calculating a ratio between the scalar of the true value loss function and the scalar of the distillation loss function; An adaptive adjustment parameter is determined according to a ratio between a scalar of the true value loss function and a scalar of the distillation loss function.

4. The method according to claim 1 or 2, characterized in that: The step of determining the adaptive adjustment parameter comprises: An adaptive adjustment parameter is determined according to a current iteration round corresponding to the training sample.

5. The method according to claim 4, characterized in that The step of determining the adaptive adjustment parameter according to the current iteration round corresponding to the training sample includes: Obtaining a preset total number of iterations and a current number of iterations corresponding to the training sample; Determine a monotone non-increasing function according to the current iteration round and the total iteration round; An adaptive adjustment parameter is determined according to the monotone non-increasing function.

6. The method according to claim 1 or 2, characterized in that: The determining of the total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameter includes: Determine a distillation loss adaptive function according to the distillation loss function and the adaptive adjustment parameter; A total loss function is determined according to the true value loss function and the distillation loss adaptive function.

7. The method according to claim 6, characterized in that The step of determining the distillation loss adaptive function according to the distillation loss function and the adaptive adjustment parameter includes: According to the distillation loss function and the adaptive adjustment parameter, calculating the product of the distillation loss function and the adaptive adjustment parameter; A distillation loss adaptive function is determined according to the product of the distillation loss function and the adaptive adjustment parameter.

8. The method according to claim 6, characterized in that Determining a total loss function according to the true value loss function and the distillation loss adaptive function includes: The true value loss function and the distillation loss adaptive function are weightedly summed to obtain a total loss function.

9. The method according to claim 1 or 2, characterized in that: The training sample is an image sample; and determining the distillation loss function according to the first output value and the second output value includes: A feature map distance between the student model and the teacher model is calculated according to the first output value and the second output value, and the feature map distance is determined as a distillation loss function.

10. An image processing method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a deep learning model to obtain a posture estimation result corresponding to the image to be processed; The deep learning model is obtained by training according to the model training method according to any one of claims 1 to 8.

11. A model training device, characterized in that: include: An input module, used to obtain training samples, and input the training samples into the student model and the teacher model respectively, obtain a first output value and a second output value, and determine a true value loss function and a distillation loss function according to the first output value, the second output value and the true value label of the training sample; A determination module, used to determine adaptive adjustment parameters; The determination module is further used to determine the total loss function according to the true value loss function, the distillation loss function and the adaptive adjustment parameter; A training module is used to train the student model according to the total loss function.

12. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A detection module is used to input the image to be processed into a deep learning model for posture estimation, and obtain a posture estimation result corresponding to the image to be processed; The deep learning model is obtained by training according to the model training method according to any one of claims 1 to 9.

13. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the model training method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the model training method according to any one of claims 1 to 10 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for model training as described in any one of claims 1 to 10 is implemented.

Citation Information

Cited By

  • Model training method and device, equipment and storage medium

    CN121257651A