Network model training method and device
By using pre-training loss value changes and two-stage training strategies in network model training, the problem of poor performance of existing data set distillation methods at low compression rates is solved, and efficient data set distillation effect under low loss situations is achieved .
Patent Information
- Application Number
- CN202411119398.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-17
AI Technical Summary
The existing data set distillation method has deteriorated effect at low compression rates, and it is impossible to achieve lossless compression of the original data set, making it difficult to effectively compress massive data under low loss conditions.
The initial network model and the number of training times are determined by the change of the loss value of the network model during pre-training, and it is divided into two stages of training: the first stage uses a sample set that cannot be recognized by the initial network model to extract complex features, and the second stage uses correct recognition The sample set and unidentified sample set are trained together to ensure that the model does not forget simple features, thereby improving the effectiveness of data set distillation at low compression rates.
It ensures the effectiveness of data set distillation at low compression rates, reduces the training time of network models, and improves the information density and compression ratio of data set distillation.
Smart Images
Figure CN119150951A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning technology, and in particular to a method and device for training a network model. Background Art
[0002] In the existing technology, training large models often requires massive amounts of data. These massive amounts of data require a lot of hardware resources during storage and training, and the cost of these hardware resources is often too high for enterprises. How to compress data with low loss is an important direction for dealing with massive data in the future. Dataset distillation aims to compress a large dataset into a small dataset in a generative way. Thanks to the high information density of the generated data, when the compression rate is high, the effect of dataset distillation far exceeds other data compression methods such as dataset pruning. Most mainstream dataset distillation methods complete feature extraction and compression by matching certain indicators of the model on the target dataset and the generated dataset. Common methods include knowledge distillation, self-distillation, and applicable networks. However, as the compression rate gradually decreases, the effect of existing dataset distillation methods gradually deteriorates, and is even weaker than randomly selecting the same amount of data from the original dataset. This results in the existing dataset distillation methods being limited to high compression rate scenarios and unable to achieve lossless compression of the original dataset. Summary of the invention
[0003] The present application provides a network model training method and device to improve the effectiveness of data set distillation through the network model.
[0004] In a first aspect, an embodiment of the present application provides a method for training a network model, comprising:
[0005] Based on the change in the loss value of the network model during the pre-training process, determine the initial network model to be trained and the number of training times N;
[0006] Identify the sample data set by using the initial network model to determine a first sample set and a second sample set, wherein the first sample set includes samples in the sample data set that are correctly identified by the initial network model, and the second sample set includes samples in the sample data set other than the first sample set;
[0007] Performing a first-stage training on the initial network model using the second sample set until the number of training times in the first stage meets the number of training times N, thereby obtaining an intermediate network model;
[0008] The intermediate network model is trained in the second stage using the sample data set until the number of training times in the second stage meets the number of training times N, thereby obtaining a target network model for data set distillation.
[0009] Based on the above scheme, when training the network model, the present application controls the initial network model and the number of training times during training by changing the loss of pre-training, thereby controlling the compression ratio of the model to the data set and the difficulty of the network model to generate data. In the first round of training, a sample set that cannot be recognized by the initial network model is used for training so that the intermediate network model can extract complex features. Furthermore, in the second round of training, the intermediate network model is trained using the first sample set and the second sample set, so that the model does not forget simple features during the training process, so that the trained network model can ensure the effectiveness of the data set distillation under low compression rates.
[0010] In a possible implementation, based on the change in the loss value of the network model during the pre-training process, determining the initial network model to be trained and the number of training times N includes:
[0011] Performing M rounds of pre-training on the network model based on the training sample set, determining a loss value corresponding to each round of pre-training, where M is greater than N;
[0012] Determine the total loss gradient according to the M loss values, use the round corresponding to the network model when the first condition is met as the first training round, and use the network model corresponding to the first training round as the initial network model, wherein the first condition is that the loss gradient is a first set proportion of the total loss gradient;
[0013] The round corresponding to the network model when the second condition is met is used as the second training round, and the number of training rounds N is determined based on the first training round and the second training round; wherein the second condition is that the loss gradient is a second set proportion of the total loss gradient, and the second set proportion is greater than the first set proportion.
[0014] Based on the above scheme, the change of loss value during pre-training is used to determine whether to start training from the initial network model, and the number of training rounds is determined according to the change of loss value. By controlling the initial network parameters and training rounds of the network model, the training time of the network model can be reduced.
[0015] In a possible implementation, the performing a second phase of training on the intermediate network model using the sample data set until the number of training times in the second phase meets the number of training times N includes:
[0016] Each training epoch does the following:
[0017] Dividing the second sample set into K batches, and interpolating the first sample set into the K batches to obtain K sample sets;
[0018] The K sample sets are input into the intermediate network model to adjust the network parameters according to the result output by the intermediate network model.
[0019] Based on the above scheme, reverse distillation is added in the training process, and the network is trained based on the later complex features and simple features to ensure the stability of the network model for feature extraction.
[0020] In a possible implementation, the first-stage training of the initial network model using the second sample set until the number of training times in the first stage meets the number of training times N to obtain an intermediate network model includes:
[0021] Each training epoch does the following:
[0022] Training the initial network model using the second sample set to obtain a target loss value;
[0023] The network parameters of the initial network model are adjusted according to the target loss value to obtain an intermediate network model.
[0024] In a possible implementation manner, the training the initial network model using the second sample set to obtain a target loss value includes:
[0025] Inputting the second sample set into the initial network model to perform feature extraction, and obtaining a predicted soft label for each sample in the second sample set;
[0026] Determine a first loss value for the predicted soft label and the sample label of each sample, and determine a second loss value according to the type of the sample label in the second sample set and the label type of the predicted soft label of each sample;
[0027] The first loss value and the second loss value are weighted to obtain a target loss value.
[0028] Based on the above scheme, the hard label is replaced with the soft label through the label smoothing method, and then the loss value is determined by the first loss value and the second loss value of the sample label, thereby improving the richness of the features extracted by the network model.
[0029] In a second aspect, an embodiment of the present application provides a network model training device, including:
[0030] A determination module, used to determine the initial network model to be trained and the number of training times N based on the change of the loss value of the network model during the pre-training process;
[0031] Identify the sample data set by using the initial network model to determine a first sample set and a second sample set, wherein the first sample set includes samples in the sample data set that are correctly identified by the initial network model, and the second sample set includes samples in the sample data set other than the first sample set;
[0032] A training module, used to perform a first-stage training on the initial network model using the second sample set until the number of training times in the first stage meets the number of training times N, thereby obtaining an intermediate network model;
[0033] The intermediate network model is trained in the second stage using the sample data set until the number of training times in the second stage meets the number of training times N, thereby obtaining a target network model for data set distillation.
[0034] In a possible implementation, the determination module, when determining the initial network model to be trained and the number of training times N based on the change in the loss value of the network model during the pre-training process, is specifically used to:
[0035] Performing M rounds of pre-training on the network model based on the training sample set, determining a loss value corresponding to each round of pre-training, where M is greater than N;
[0036] Determine the total loss gradient according to the M loss values, use the round corresponding to the network model when the first condition is met as the first training round, and use the network model corresponding to the first training round as the initial network model, wherein the first condition is that the loss gradient is a first set proportion of the total loss gradient;
[0037] The round corresponding to the network model when the second condition is met is used as the second training round, and the number of training rounds N is determined based on the first training round and the second training round; wherein the second condition is that the loss gradient is a second set proportion of the total loss gradient, and the second set proportion is greater than the first set proportion.
[0038] In a possible implementation, the training module, when performing the second-stage training on the intermediate network model using the sample data set until the number of training times in the second stage meets the number of training times N, is specifically configured to:
[0039] Each training epoch does the following:
[0040] Dividing the second sample set into K batches, and interpolating the first sample set into the K batches to obtain K sample sets;
[0041] The K sample sets are input into the intermediate network model to adjust the network parameters according to the result output by the intermediate network model.
[0042] In a possible implementation, the training module, when performing the first-stage training on the initial network model through the second sample set until the number of training times in the first stage meets the number of training times N, and obtaining the intermediate network model, is specifically configured to:
[0043] Each training epoch does the following:
[0044] Training the initial network model using the second sample set to obtain a target loss value;
[0045] The network parameters of the initial network model are adjusted according to the target loss value to obtain an intermediate network model.
[0046] In a possible implementation, when the training module trains the initial network model through the second sample set to obtain the target loss value, it is specifically used to:
[0047] Inputting the second sample set into the initial network model to perform feature extraction, and obtaining a predicted soft label for each sample in the second sample set;
[0048] Determine a first loss value for the predicted soft label and the sample label of each sample, and determine a second loss value according to the type of the sample label in the second sample set and the label type of the predicted soft label of each sample;
[0049] The first loss value and the second loss value are weighted to obtain a target loss value.
[0050] In a third aspect, an embodiment of the present application provides an execution device, including:
[0051] A memory for storing program instructions;
[0052] The processor is used to obtain the program instructions in the memory and implement the method described in the first aspect and different implementation methods of the first aspect according to the obtained program instructions.
[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes computer instructions. When the computer instructions are executed by a computer, the method described in the first aspect and different implementation methods of the first aspect are implemented.
[0054] In addition, the technical effects brought about by any implementation method of the second to fourth aspects can refer to the technical effects brought about by the first aspect and different implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0056] Figure 1 A schematic diagram of the system architecture provided for an embodiment of the present application;
[0057] Figure 2 A schematic diagram of the server structure provided in the embodiment of the present application;
[0058] Figure 3 A flowchart of a network model training method provided in an embodiment of the present application;
[0059] Figure 4 A schematic diagram of a network model training device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0061] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0062] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0063] In the existing technology, training large models often requires massive amounts of data. These massive amounts of data require a lot of hardware resources during storage and training, and the cost of these hardware resources is often too high for enterprises. How to compress data with low loss is an important direction for dealing with massive data in the future. Dataset distillation aims to compress a large dataset into a small dataset in a generative way. Thanks to the high information density of the generated data, when the compression rate is high, the effect of dataset distillation far exceeds other data compression methods such as dataset pruning. Most mainstream dataset distillation methods complete feature extraction and compression by matching certain indicators of the model on the target dataset and the generated dataset. Common methods include knowledge distillation, self-distillation, and applicable networks. However, as the compression rate gradually decreases, the effect of existing dataset distillation methods gradually deteriorates, and is even weaker than randomly selecting the same amount of data from the original dataset. This results in the existing dataset distillation methods being limited to high compression rate scenarios and unable to achieve lossless compression of the original dataset.
[0064] In response to the above problems, an embodiment of the present application provides a method and device for training a network model. When training a network model, the initial network model and the number of training times during training are controlled by changing the loss of pre-training, thereby controlling the compression ratio of the model to the data set and the difficulty of the network model to generate data. In the first round of training, a sample set that cannot be recognized by the initial network model is used for training, so that the intermediate network model can extract complex features. Furthermore, in the second round of training, the intermediate network model is trained using the first sample set and the second sample set, so that the model does not forget simple features during the training process, so that the trained network model can ensure the effectiveness of the data set distillation under low compression rates.
[0065] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.
[0066] The training method of the network model provided in the embodiment of the present application can be implemented by an execution device. In some embodiments, the execution device can be a terminal device. The terminal device can be a display device with a display function. The display device can include: a smart TV, a mobile phone, a tablet computer, etc. In other embodiments, the execution device can be an electronic device, and the electronic device can be implemented by one or more servers, and the server can be a local server or a cloud server.
[0067] The following takes the execution device as an electronic device as an example to describe the structure and application scenario of the execution device. The electronic device can be implemented by one or more servers. Figure 1As shown, the server 100 can be implemented by a physical server or a virtual server. The server can be implemented by a single server or a server cluster composed of multiple servers. The training method of the network model provided in this application can be implemented by a single server or a server cluster. Figure 1 In the example, the server 100 is connected to the terminal device 200. The server 100 can execute the training method of the network model. In some scenarios, the server 100 can also receive the data set distillation task sent by the terminal device 200, or send the data set distillation result to the terminal device 200.
[0068] In some scenarios, the electronic device may also be a personal computer, a handheld or laptop device, a personal digital assistant, etc.
[0069] As an example, see Figure 2 As shown, the electronic device may include a processor 110 and a communication interface 120. The electronic device may also include a memory 130. Of course, the electronic device may also include other components. Figure 2 Not shown in FIG.
[0070] The communication interface 110 is used to communicate with the display device, to receive the training task and the data set distillation task of the network model sent by the terminal device 200, or to send the data set distillation effect to the terminal device 200.
[0071] In the embodiment of the present application, the processor 110 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0072] The processor 110 is the control center of the electronic device, which connects various parts of the electronic device using various interfaces and routes, executes various functions of the electronic device and processes data by running or executing software programs / modules stored in the memory 130, and calling data stored in the memory 130. Optionally, the processor 110 may include one or more processing units. The processor 110 may be, for example, a control component such as a processor, a microprocessor, a controller, and may be, for example, a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.
[0073] The memory 130 can be used to store software programs and modules. The processor 110 executes various functional applications and data processing by running the software programs and modules stored in the memory 130. The memory 130 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to business processing, etc. The memory 130, as a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 130 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Prom), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 130 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 130 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0074] It should be noted that the above Figure 1-2 The structure shown is only an example and is not limited to this embodiment of the present invention.
[0075] The present application embodiment provides a method for training a network model. Figure 3 The process of the training method of the network model is exemplarily shown, and the process can be executed by an execution device, which can be Figure 2 The electronic device shown can specifically execute the training of the network model through the processor 110 in the electronic device.
[0076] The specific process is as follows:
[0077] Step S301, based on the change in the loss value of the network model during the pre-training process, determine the initial network model to be trained and the number of training times N.
[0078] In some embodiments, when determining the initial network model to be trained and the number of training times N based on the change in the loss value of the network model during the pre-training process, it can be determined in the following manner: pre-train the network model for M rounds based on the training sample set, and determine the loss value corresponding to each round of pre-training. Wherein, M is greater than N. Then, the total loss gradient is determined based on the M loss values, and the round corresponding to the network model when the first condition is met is used as the first training round, and the network model corresponding to the first training round is used as the initial network model. Wherein, the first condition is that the loss gradient is the first set proportion of the total loss gradient. Furthermore, the round corresponding to the network model when the second condition is met is used as the second training round, and the number of training times N is determined based on the first training round and the second training round; wherein, the second condition is that the loss gradient is the second set proportion of the total loss gradient, and the second set proportion is greater than the first set proportion.
[0079] Specifically, during the training process, since the number of training rounds increases the complexity of the model, the difficulty of generating features can be controlled by controlling the training trajectory used for matching. As an example, J can be used to represent the pre-training trajectory of the network model, and θ t represents the parameters of the network model during the tth round of pre-training, then the training trajectory of the pre-training of the network model can be expressed as:
[0080] J={θ0,……,θ L-1 ,θ L , ..., θ H ,θ H+1 , ..., θ n}
[0081] As shown in the above formula, where θ0 to θ L-1 Determined to be the initial stage of training, during which the network model mostly generates simple features. θ H+1 to θn For the late stage of pre-training, most of them generate difficult features. L to θ H is the intercepted training trajectory, at θ L to θ H Data distillation is performed between parameters.
[0082] Among them, θ L and θ H The parameters are determined according to the fitting effect obtained by the training sample set, that is, determined by the loss value of the training sample set.
[0083] Specifically, θ L and θ H The parameters satisfy the conditions shown in the following formula:
[0084] θ L ~o(1-Loss)*A;
[0085] θ H ~o(1-Loss)*B;
[0086] Among them, o(1-Loss) represents the total loss gradient, A is the first setting proportion, B is the second setting proportion, and B is greater than A.
[0087] From the above formula, we know that θ L is the network parameter of the network model when the loss gradient reaches the first set proportion of the total loss gradient, then θ L The corresponding network model is used as the initial network model, and the training round corresponding to the initial network model is used as the first training round. Similarly, θ H is the network parameter of the network model when the loss gradient reaches the second set proportion of the total loss gradient, then the network parameter is θ H The training round corresponding to the network model is the second training round. Further, the number of training times N can be determined according to the first training round and the second training round.
[0088] As an example, A can be 25% and B can be 75%. Then the first training round is the training round when the loss reaches the 25% percentile, and the second training round is the training round when the loss reaches the 75% percentile.
[0089] Step S302: Identify the sample data set through the initial network model to determine a first sample set and a second sample set.
[0090] The first sample set includes samples in the sample data set that are correctly identified by the initial network model, and the second sample set includes samples in the sample data set except the first sample set.
[0091] As an example, the sample data set can be input into the initial network model to determine the samples correctly classified by the initial network model as the first sample set. Furthermore, the samples in the sample data set that are not in the first sample set can be used as the second sample set.
[0092] Step S303, the initial network model is trained in the first stage by using the second sample set until the number of training times in the first stage meets the number of training times N, thereby obtaining an intermediate network model.
[0093] In some embodiments, when the initial network model is trained in the first stage by using the second sample set until the number of training times in the first stage meets the number of training times N, and an intermediate network model is obtained, it can be specifically implemented in the following manner:
[0094] The following operations are performed in each training round: the initial network model is trained through the second sample set to obtain a target loss value; the network parameters of the initial network model are adjusted according to the target loss value to obtain an intermediate network model.
[0095] Specifically, the target loss value can be determined in the following manner: the second sample set is input into the initial network model for feature extraction to obtain the predicted soft label of each sample in the second sample set. Further, a first loss value is determined for the predicted soft label and sample label of each sample, and a second loss value is determined according to the type of sample label in the second sample set and the type of label of the predicted soft label of each sample. The first loss value and the second loss value are then weighted to obtain the target loss value, so as to train the initial network model with the target loss value to obtain the intermediate network model.
[0096] As an example, in order to make the distilled data set contain more information, the label smoothing method can be used to replace the hard label (one-hot label) of the distilled data with a soft label, and set the smoothing index s. The smoothing index s is used to represent the number of predicted label types. During the training process, s is continuously adjusted to make the smoothing index s closer to the number of sample label types in the sample data set.
[0097] When the initial network model is trained in the first stage through the second sample set, in the early stage, the network parameters can only extract simple features, but as the number of training times increases, the network model can extract complex features.
[0098] Step S304, the intermediate network model is trained in the second stage using the sample data set until the number of training times in the second stage meets the number of training times N, thereby obtaining a target network model for data set distillation.
[0099] In some embodiments, each training round performs the following operations: the second sample set is divided into K batches, and the first sample set is interpolated into the K batches to obtain K sample sets. Further, the K sample sets are input into the intermediate network model to adjust the network parameters according to the results output by the intermediate network model.
[0100] Furthermore, the dataset can be distilled through the trained target network model to obtain a distilled dataset.
[0101] In the present application, when performing the second stage of training, by interpolating the first sample set into the second sample set, it is possible to enrich the features of the data set without affecting the stability of the features.
[0102] In the present application, based on the pre-training trajectory of the network model, it can be known that the network model can extract certain features in the data set at one time. In the early stage of training, the loss drops very quickly, that is, the network parameters learn simple features very quickly. But in the later stage of training, the loss drops very slowly. The network model learns complex features very slowly in the later stage. Based on this, the present application determines the initial network model and the number of training times of the network model based on the changes in the pre-training trajectory and the loss value to control the difficulty of generating features. Furthermore, through two-stage training, the network model can extract effective features from the sample data set, thereby ensuring the effectiveness of the data set distillation during use. Through the two-stage training method, even under low compression rates, the effectiveness of the network model for the data after data set distillation can be guaranteed.
[0103] Based on the same technical concept, see Figure 4 As shown, Figure 4 The present application provides a network model training device 400, which can implement any step in the above network model training method. To avoid repetition, it is not described here. The device 400 includes a determination module 401 and a training module 402.
[0104] A determination module 401 is used to determine an initial network model to be trained and a number of training times N based on a change in a loss value of the network model during a pre-training process;
[0105] Identify the sample data set by using the initial network model to determine a first sample set and a second sample set, wherein the first sample set includes samples in the sample data set that are correctly identified by the initial network model, and the second sample set includes samples in the sample data set other than the first sample set;
[0106] A training module 402 is used to perform a first-stage training on the initial network model using the second sample set until the number of training times in the first stage meets the number of training times N, thereby obtaining an intermediate network model;
[0107] The intermediate network model is trained in the second stage using the sample data set until the number of training times in the second stage meets the number of training times N, thereby obtaining a target network model for data set distillation.
[0108] In a possible implementation, the determination module 401, when determining the initial network model to be trained and the number of training times N based on the change in the loss value of the network model during the pre-training process, is specifically used to:
[0109] Performing M rounds of pre-training on the network model based on the training sample set, determining a loss value corresponding to each round of pre-training, where M is greater than N;
[0110] Determine the total loss gradient according to the M loss values, use the round corresponding to the network model when the first condition is met as the first training round, and use the network model corresponding to the first training round as the initial network model, wherein the first condition is that the loss gradient is a first set proportion of the total loss gradient;
[0111] The round corresponding to the network model when the second condition is met is used as the second training round, and the number of training rounds N is determined based on the first training round and the second training round; wherein the second condition is that the loss gradient is a second set proportion of the total loss gradient, and the second set proportion is greater than the first set proportion.
[0112] In a possible implementation, the training module 402, when performing the second-stage training on the intermediate network model using the sample data set until the number of training times in the second stage meets the number of training times N, is specifically configured to:
[0113] Each training epoch does the following:
[0114] Dividing the second sample set into K batches, and interpolating the first sample set into the K batches to obtain K sample sets;
[0115] The K sample sets are input into the intermediate network model to adjust the network parameters according to the result output by the intermediate network model.
[0116] In a possible implementation, the training module 402, when performing the first phase training on the initial network model through the second sample set until the number of training times in the first phase meets the number of training times N, and obtaining the intermediate network model, is specifically configured to:
[0117] Each training epoch does the following:
[0118] Training the initial network model using the second sample set to obtain a target loss value;
[0119] The network parameters of the initial network model are adjusted according to the target loss value to obtain an intermediate network model.
[0120] In a possible implementation, the training module 402, when training the initial network model through the second sample set to obtain a target loss value, is specifically configured to:
[0121] Inputting the second sample set into the initial network model to perform feature extraction, and obtaining a predicted soft label for each sample in the second sample set;
[0122] Determine a first loss value for the predicted soft label and the sample label of each sample, and determine a second loss value according to the type of the sample label in the second sample set and the label type of the predicted soft label of each sample;
[0123] The first loss value and the second loss value are weighted to obtain a target loss value.
[0124] Based on the same technical concept, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes any step in the above-mentioned network model training method.
[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0127] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A network model training method, characterized in that: include: Based on the change in the loss value of the network model during the pre-training process, determine the initial network model to be trained and the number of training times N; Identify the sample data set by using the initial network model to determine a first sample set and a second sample set, wherein the first sample set includes samples in the sample data set that are correctly identified by the initial network model, and the second sample set includes samples in the sample data set other than the first sample set; Performing a first-stage training on the initial network model using the second sample set until the number of training times in the first stage meets the number of training times N, thereby obtaining an intermediate network model; The intermediate network model is trained in the second stage using the sample data set until the number of training times in the second stage meets the number of training times N, thereby obtaining a target network model for data set distillation.
2. The method according to claim 1, characterized in that Based on the change in the loss value of the network model during the pre-training process, the initial network model to be trained and the number of training times N are determined, including: Performing M rounds of pre-training on the network model based on the training sample set, determining a loss value corresponding to each round of pre-training, where M is greater than N; Determine the total loss gradient according to the M loss values, use the round corresponding to the network model when the first condition is met as the first training round, and use the network model corresponding to the first training round as the initial network model, wherein the first condition is that the loss gradient is a first set proportion of the total loss gradient; The round corresponding to the network model when the second condition is met is used as the second training round, and the number of training rounds N is determined based on the first training round and the second training round; wherein the second condition is that the loss gradient is a second set proportion of the total loss gradient, and the second set proportion is greater than the first set proportion.
3. The method according to claim 1, characterized in that The performing the second phase of training on the intermediate network model by using the sample data set until the number of training times in the second phase meets the number of training times N, comprises: Each training epoch does the following: Dividing the second sample set into K batches, and interpolating the first sample set into the K batches to obtain K sample sets; The K sample sets are input into the intermediate network model to adjust the network parameters according to the result output by the intermediate network model.
4. The method according to claim 1, characterized in that The first phase of training the initial network model using the second sample set until the number of training times in the first phase meets the number of training times N, to obtain an intermediate network model, includes: Each training epoch does the following: Training the initial network model using the second sample set to obtain a target loss value; The network parameters of the initial network model are adjusted according to the target loss value to obtain an intermediate network model.
5. The method according to claim 4, characterized in that The training of the initial network model by using the second sample set to obtain a target loss value includes: Inputting the second sample set into the initial network model to perform feature extraction, and obtaining a predicted soft label for each sample in the second sample set; Determine a first loss value for the predicted soft label and the sample label of each sample, and determine a second loss value according to the type of the sample label in the second sample set and the label type of the predicted soft label of each sample; The first loss value and the second loss value are weighted to obtain a target loss value.
6. A network model training device, characterized in that: include: A determination module, used to determine the initial network model to be trained and the number of training times N based on the change of the loss value of the network model during the pre-training process; Identify the sample data set by using the initial network model to determine a first sample set and a second sample set, wherein the first sample set includes samples in the sample data set that are correctly identified by the initial network model, and the second sample set includes samples in the sample data set other than the first sample set; A training module, used to perform a first-stage training on the initial network model using the second sample set until the number of training times in the first stage meets the number of training times N, thereby obtaining an intermediate network model; The intermediate network model is trained in the second stage using the sample data set until the number of training times in the second stage meets the number of training times N, thereby obtaining a target network model for data set distillation.
7. The device according to claim 6, characterized in that The determination module, when determining the initial network model to be trained and the number of training times N based on the change in the loss value of the network model during the pre-training process, is specifically used to: Performing M rounds of pre-training on the network model based on the training sample set, determining a loss value corresponding to each round of pre-training, where M is greater than N; Determine the total loss gradient according to the M loss values, use the round corresponding to the network model when the first condition is met as the first training round, and use the network model corresponding to the first training round as the initial network model, wherein the first condition is that the loss gradient is a first set proportion of the total loss gradient; The round corresponding to the network model when the second condition is met is used as the second training round, and the number of training rounds N is determined based on the first training round and the second training round; wherein the second condition is that the loss gradient is a second set proportion of the total loss gradient, and the second set proportion is greater than the first set proportion.
8. The device according to claim 6, characterized in that The training module, when performing the second-stage training on the intermediate network model through the sample data set until the number of training times in the second stage meets the number of training times N, is specifically used to: Each training epoch does the following: Dividing the second sample set into K batches, and interpolating the first sample set into the K batches to obtain K sample sets; The K sample sets are input into the intermediate network model to adjust the network parameters according to the result output by the intermediate network model.
9. An execution device, characterized in that: include: A memory for storing program instructions; A processor is used to obtain program instructions in the memory and implement the method according to any one of claims 1 to 5 according to the obtained program instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed by a computer, the method according to any one of claims 1 to 5 is implemented.