Model training methods, devices, and electronic equipment based on self-network knowledge distillation

Abstract

A model training method, apparatus, and electronic device based on self-network knowledge distillation are disclosed. The method includes: determining a hybrid image of at least two training images; processing the at least two training images and the hybrid image using a neural network model to obtain a processing result; determining the model loss value of the neural network model based on the processing result; and training the neural network model based on the model loss value. Embodiments of this disclosure can significantly reduce training overhead.

Description

Technical Field

[0001] This disclosure relates to neural network model technology, and in particular to a model training method, apparatus, and electronic device based on self-network knowledge distillation. Background Technology

[0002] Current knowledge distillation techniques typically use a teacher-student model-based training framework, which requires pre-training a large teacher network and then using the knowledge from the teacher network as supervisory information to assist in the training of a small student network. Summary of the Invention

[0003] To address the technical problem of high training overhead caused by the two-stage training process required by current knowledge distillation techniques, this disclosure is proposed. Embodiments of this disclosure provide a model training method, apparatus, and electronic device based on self-network knowledge distillation.

[0004] According to one aspect of the embodiments of this disclosure, a model training method based on self-network knowledge distillation is provided, comprising:

[0005] Determine a blended image of at least two training frames;

[0006] The at least two training images and the mixed image are processed using a neural network model to obtain the processing result;

[0007] Based on the processing results, the model loss value of the neural network model is determined;

[0008] The neural network model is trained based on the model loss value.

[0009] According to another aspect of the present disclosure, a model training apparatus based on self-network knowledge distillation is provided, comprising:

[0010] The first determining module is used to determine a blended image of at least two training images;

[0011] The processing module is used to process the at least two training images and the mixed image determined by the first determining module through a neural network model to obtain a processing result;

[0012] The second determining module is used to determine the model loss value of the neural network model based on the processing result obtained by the processing module.

[0013] The training module is used to train the neural network model based on the model loss value determined by the second determining module.

[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for executing the above-described model training method based on self-network knowledge distillation.

[0015] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0016] processor;

[0017] Memory used to store the processor's executable instructions;

[0018] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-described model training method based on self-network knowledge distillation.

[0019] Based on the model training method, apparatus, computer-readable storage medium, and electronic device based on self-network knowledge distillation provided in the above embodiments of this disclosure, at least two training images and a mixed image of at least two training images can be processed by a neural network model. Based on the processing results, the model loss value of the neural network model can be determined so that the neural network model can be trained based on the model loss value. Thus, the embodiments of this disclosure can use training images and mixed images to guide the knowledge distillation of a single neural network model, thereby simplifying the training process into an end-to-end process without the need for pre-training a teacher network, thus significantly reducing training overhead.

[0020] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 This is a schematic flowchart of a model training method based on self-network knowledge distillation provided in an exemplary embodiment of this disclosure.

[0023] Figure 2 This is a schematic diagram illustrating the training principle of a model based on self-network knowledge distillation in an exemplary embodiment of this disclosure.

[0024] Figure 3 This is a flowchart illustrating a model training method based on self-network knowledge distillation, provided in another exemplary embodiment of this disclosure.

[0025] Figure 4 This is a flowchart illustrating a model training method based on self-network knowledge distillation, provided in yet another exemplary embodiment of this disclosure.

[0026] Figure 5 This is a flowchart illustrating a model training method based on self-network knowledge distillation, provided in yet another exemplary embodiment of this disclosure.

[0027] Figure 6 This is a schematic diagram of the structure of a model training apparatus based on self-network knowledge distillation provided in an exemplary embodiment of this disclosure.

[0028] Figure 7 This is a schematic diagram of the structure of a model training apparatus based on self-network knowledge distillation provided in another exemplary embodiment of this disclosure.

[0029] Figure 8 This is a schematic diagram of the structure of a model training apparatus based on self-network knowledge distillation provided in another exemplary embodiment of this disclosure.

[0030] Figure 9 This is a schematic diagram of the structure of a model training device based on self-network knowledge distillation, provided in yet another exemplary embodiment of this disclosure.

[0031] Figure 10 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0032] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0033] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0034] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0035] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0036] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0037] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0038] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0039] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0040] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0041] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0042] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0043] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0044] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0045] Application Overview

[0046] Knowledge distillation is an important model compression method. Current knowledge distillation techniques typically use a teacher-student-based training framework. This framework requires pre-training a large, complex, highly capable, and accurate teacher network, and then using the knowledge from the teacher network as supervisory information to assist in training a small, concise student network suitable for inference deployment.

[0047] It should be noted that the training framework based on the teacher-student model can transfer the knowledge learned by the teacher network to the student network, thereby enhancing the generalization ability of the student network. However, this training framework requires a two-stage training process, resulting in high training costs.

[0048] Exemplary System

[0049] To address the technical issues mentioned in the "Application Overview" section above, the embodiments of this disclosure do not employ knowledge distillation technology based on a teacher-student training framework. Instead, they employ self-network knowledge distillation technology. Self-network knowledge distillation technology does not require pre-training a teacher network but relies on the internal knowledge of a single network to improve performance, thereby simplifying the training process into an end-to-end process and significantly reducing training overhead.

[0050] Exemplary methods

[0051] Figure 1 This is a schematic flowchart of a model training method based on self-network knowledge distillation provided in an exemplary embodiment of this disclosure. Figure 1 The method shown can be applied to electronic devices. Figure 1 The method shown may include steps 110, 120, 130 and 140, which are described below.

[0052] Step 110: Determine a blended image of at least two training images.

[0053] Before step 110, at least two training images (also known as original images) can be randomly obtained from the image database. All training images in the at least two training images can have the same image size.

[0054] In step 110, image blending techniques can be used to blend at least two training images to obtain a blended image.

[0055] Assume that there are at least two training images, specifically two training images, denoted as x. i and x j The mixed image is represented as Then x i x j , The following formula can be satisfied:

[0056]

[0057] Where λ can be a random mixing coefficient, and λ can follow a beta distribution, x i x j , The effects can be seen in Figure 2 .

[0058] Step 120: Process at least two training images and the mixed image using a neural network model to obtain the processing result.

[0059] Optionally, the neural network model can be a convolutional neural network model, which can be used for image classification tasks. Of course, the neural network model can also be used for other image-related tasks. For ease of understanding, the embodiments of this disclosure all use the case of using a neural network model for image classification tasks as an example for illustration.

[0060] In step 120, at least two training images and a mixed image can be provided as input to the neural network model. The neural network model can process each training image and the mixed image in the at least two training images to obtain the processing result.

[0061] Step 130: Based on the processing results, determine the model loss value of the neural network model.

[0062] In step 130, based on the processing results, the model loss value of the neural network model can be determined using a preset loss function.

[0063] Optionally, the preset loss functions include, but are not limited to, the Mean Absence Error (MAE) loss function, the Mean Square Error (MSE) loss function, the cross-entropy loss function, etc., which will not be listed here. Among them, the Mean Absence Error loss function can also be called the L1 loss function; the Mean Square Error loss function can also be called the L2 loss function.

[0064] Step 140: Train the neural network model based on the model loss value.

[0065] In step 140, the model parameters of the neural network model can be adjusted based on the model loss value using the stochastic gradient descent method until the neural network model converges, thereby completing the training of the neural network model.

[0066] In the embodiments of this disclosure, at least two training images and a mixed image of at least two training images can be processed by a neural network model. Based on the processing results, the model loss value of the neural network model can be determined so that the neural network model can be trained based on the model loss value. Thus, the embodiments of this disclosure can use training images and mixed images to guide the knowledge distillation of a single neural network model, thereby simplifying the training process into an end-to-end process without the need for pre-training the teacher network, thus significantly reducing training overhead.

[0067] exist Figure 1 Based on the illustrated embodiments, as Figure 3 As shown, step 120 includes step 1201.

[0068] Step 1201: Generate at least two first feature maps for at least two training images through a predetermined sub-network in the neural network model, and generate a second feature map for the mixed image. The processing result includes at least two first feature maps and a second feature map.

[0069] Optionally, the neural network model may include a feature extractor and a classifier; wherein the feature extractor is used to perform feature extraction processing on the input image to obtain the feature extraction result; and the classifier is used to obtain the image classification result based on the feature extraction result of the feature extractor.

[0070] If the predetermined subnetwork includes a feature extractor, the feature extractor can process each of the at least two training images to obtain a first feature map representing the feature extraction result, thereby obtaining at least two first feature maps corresponding one-to-one with the at least two training images. Similarly, the feature extractor can process the mixed image to obtain a second feature map representing the feature extraction result.

[0071] If the predetermined sub-network includes a classifier, the classifier can process the feature extraction results corresponding to each of the at least two training images to obtain a first feature map representing the image classification result, thereby obtaining at least two first feature maps that correspond one-to-one with the at least two training images. Similarly, the classifier can process the feature extraction results corresponding to the mixed images to obtain a second feature map representing the image classification result.

[0072] Step 130 includes steps 1301 and 1303.

[0073] Step 1301: Determine the first mixed feature map of at least two frames of first feature maps.

[0074] In step 1301, image mixing technology can be used to mix at least two frames of the first feature map to obtain a first mixed feature map.

[0075] Assuming the predefined subnetwork includes a feature extractor, at least two frames of the first feature map are specifically two frames of the first feature map and are respectively represented as F i and F j The first mixed feature map is represented as F ij Then F i F j F ij The following formula can be satisfied:

[0076] F ij =λx i +(1-λ)x j

[0077] Assuming the predefined sub-network includes a classifier, at least two frames of the first feature map are specifically two frames of the first feature map and are respectively represented as P i and P j The first mixed feature map is represented as P. ij Then P i P j P ij The following formula can be satisfied:

[0078] P ij =λP i +(1-λ)P j

[0079] Step 1303: Determine the model loss value of the neural network model by comparing the second feature map with the first mixed feature map.

[0080] In one specific implementation, the predetermined subnetwork includes: a classifier in a neural network model;

[0081] Step 1303 includes:

[0082] Based on the second feature map, determine the probability distribution of the first category;

[0083] Based on the first mixed feature map, determine the probability distribution of the second category;

[0084] Calculate the first relative entropy of the first category probability distribution with respect to the second category probability distribution;

[0085] Calculate the second relative entropy of the second category probability distribution with respect to the first category probability distribution;

[0086] Calculate the first sum of the first relative entropy and the second relative entropy;

[0087] Based on the first sum, the model loss value of the neural network model is determined.

[0088] It is understandable that relative entropy can also be called KL divergence (Kullback-Leibler divergence), which is an asymmetric measure of the difference between two probability distributions.

[0089] It should be noted that the image classification task in the embodiments of this disclosure may involve N preset categories, where N can be 2, 3, 4, or an integer greater than 4. The N preset categories include, but are not limited to, Figure 2 The birds, cats, ships, dogs, etc.

[0090] It should be noted that the second feature map may contain N probability values ​​for each pixel corresponding to N preset categories. The probability value of any pixel corresponding to any preset category represents the probability that the classifier predicts that the pixel belongs to that preset category. The N probability values ​​corresponding to any pixel can form the category probability data for that pixel. The category probability data corresponding to all pixels can form a category probability distribution. In this way, the second feature map can represent a category probability distribution, which can be used as the first category probability distribution. Similarly, the second category probability distribution can be determined based on the first mixed feature map.

[0091] Suppose the probability distribution of the first category is represented as The second category probability distribution is represented as p ij Then the first relative entropy of the first category probability distribution with respect to the second category probability distribution can be expressed as: The second relative entropy of the second category probability distribution with respect to the first category probability distribution can be expressed as: Suppose that the first sum of the first relative entropy and the second relative entropy is represented as L probability Then we have:

[0092]

[0093] Optionally, the first sum can be directly used as the model loss value of the neural network model; or, the first sum can be used as a component of the model loss value of the neural network model, and the first sum and other components of the model loss value (such as L in the following text) can be calculated separately. feature L task After that, the model loss value of the neural network model is obtained by summing.

[0094] In this implementation, the first category probability distribution determined based on the second feature map can characterize the category probability distribution generated by the neural network model for the mixed image, and the second category probability distribution determined based on the first mixed feature map can characterize the category probability distribution obtained by aggregating the image classification results of at least two training images. Based on these two category probability distributions, KL divergence calculation is performed, and then combined with addition operation, the model loss value of the neural network model can be obtained efficiently and reliably. Furthermore, using the obtained model loss value for training the neural network model can make the category probability distribution obtained by aggregating the image classification results of at least two training images as similar as possible to the category probability distribution generated for the mixed image, thereby enabling bidirectional knowledge transfer at the probability level.

[0095] In another specific implementation, the predetermined subnetwork includes: a feature extractor in a neural network model;

[0096] Step 1303 includes:

[0097] Calculate the mean squared error between the second feature map and the first mixed feature map;

[0098] The model loss value of the neural network model is determined based on the mean squared error value.

[0099] Suppose the second feature map is represented as The first mixed feature map is represented as F. ij The mean squared error between the second feature map and the first mixed feature map is denoted as L. feature Then F ij , L feature The following formula can be satisfied:

[0100]

[0101] Optionally, the mean squared error (MSE) can be directly used as the model loss value of the neural network model; or, the MSE can be used as a component of the model loss value of the neural network model, and calculated separately from other components of the model loss value (such as L mentioned above). probability L in the following text taskAfter that, the model loss value of the neural network model is obtained by summing the results.

[0102] In this implementation, by introducing the L2 loss function, the model loss value of the neural network model can be obtained efficiently and reliably. Furthermore, the second feature map can characterize the feature map generated by the neural network model performing feature extraction on the mixed image, and the first mixed feature map can characterize the feature map obtained by aggregating the feature extraction results of at least two training images after the neural network model performs feature extraction on each of the at least two training images. Using the obtained model loss value for training the neural network model can make the feature map obtained by aggregating the feature extraction results of at least two training images as similar as possible to the feature map generated by performing feature extraction on the mixed image, thereby enabling bidirectional knowledge transfer at the feature level.

[0103] In the embodiments of this disclosure, a first feature map is generated for at least two training images by a predetermined sub-network in the neural network model, and a second feature map is generated for the mixed image. A first mixed feature map of the at least two first feature maps is determined, and the model loss value of the neural network model is determined by comparing the second feature map with the first mixed feature map. This is equivalent to mining additional knowledge across images (i.e. mining additional knowledge based on at least two training images and the mixed image), and the mined knowledge is used for training the neural network model. This can capture the relationship dependencies between different images, improve the richness and diversity of knowledge used during model training, thereby improving the performance of the trained neural network model and further reducing training overhead.

[0104] exist Figure 1 Based on the illustrated embodiments, as Figure 4 As shown, step 120 includes step 1203.

[0105] Step 1203: Generate at least two third feature maps for at least two training images using a classifier in the neural network model, and generate a fourth feature map for the mixed image. The processing result includes at least two third feature maps and a fourth feature map.

[0106] In step 1203, the classifier can process the feature extraction results corresponding to each of the at least two training images to obtain a third feature map representing the image classification result, thereby obtaining at least two third feature maps that correspond one-to-one with the at least two training images. Similarly, the classifier can also process the feature extraction results corresponding to the mixed image to obtain a fourth feature map representing the image classification result.

[0107] Step 130 includes steps 1305, 1307 and 1309.

[0108] Step 1305: Determine the second mixed feature map of at least two frames of third feature maps.

[0109] It should be noted that the specific implementation of step 1305 can be referred to the description of step 1301, and will not be repeated here.

[0110] Step 1307: Determine the category label data of the mixed image based on the category label data of at least two training images.

[0111] It should be noted that the category label data of any training image may include: the category label information of each pixel in the training image; wherein, the category label information may be in one-hot encoded form. For example, the N preset categories involved in the image classification task are specifically three preset categories, namely category 1, category 2, and category 3. If the true category of a pixel in any training image is category 1, then the category label information of that pixel in the category label data of the training image can be represented as (1, 0, 0). If the true category of another pixel in the training image is category 2, then the category label information of that pixel in the category label data of the training image can be represented as (0, 1, 0). If the true category of yet another pixel in the training image is category 3, then the category label information of that pixel in the category label data of the training image can be represented as (0, 0, 1).

[0112] Given that the class label data of at least two training images are known, the class label data of the mixed image can be determined by interpolation of the class label data of at least two training images.

[0113] Assuming at least two training images, specifically two training images, the class label data of these two training images are represented as y. i and y j The category label data of the mixed images is represented as Then y i y j , The following formula can be satisfied:

[0114]

[0115] In one example, in the two training images mentioned above, the class label information of the top-left pixel of one training image is (1, 0, 0), and the class label information of the top-left pixel of the other training image is (0, 0, 1), with λ being 0.4. Then, according to the above formula, the class label information of the top-left pixel of the blended image is (0.4, 0, 0.6). In another example, in the two training images mentioned above, the class label information of the top-right pixel of one training image is (0, 1, 0), and the class label information of the top-right pixel of the other training image is (1, 0, 0), with λ being 0.5. Then, according to the above formula, the class label information of the top-right pixel of the blended image is (0.5, 0.5, 0). After obtaining the class label information of each pixel in the blended image according to the above formula, these class label information can be used to form the class label data of the blended image.

[0116] Step 1309: Based on at least two frames of third feature maps, second mixed feature maps, class label data of each of the at least two frames of training images, and class label data of the mixed images, determine the model loss value of the neural network model.

[0117] In one specific embodiment, step 1309 includes:

[0118] For each training image in at least two training images, a third category probability distribution is determined based on the third feature map corresponding to that training image.

[0119] Based on the second mixed feature map, determine the probability distribution of the fourth category;

[0120] For each training image in at least two training images, the first cross-entropy is calculated based on the third category probability distribution corresponding to the training image and the category label data of the training image;

[0121] The second cross-entropy is calculated based on the fourth category probability distribution and the category label data of the mixed images;

[0122] Calculate the second sum of at least two first cross-entropies and second cross-entropies corresponding to at least two training images;

[0123] Based on the second sum, the model loss value of the neural network model is determined.

[0124] Understandably, cross-entropy can be used to measure the difference between two probability distributions.

[0125] It should be noted that the methods for determining the probability distributions of the third and fourth categories are the same as those for determining the probability distribution of the first category mentioned above, and will not be repeated here.

[0126] It should be noted that, since the category label information in the category label data is in one-hot encoded form, the category label data of each training image can also represent a category probability distribution, and the category label data of the mixed image can also represent a category probability distribution. Thus, for any training image, cross-entropy can be calculated based on the third category probability distribution corresponding to that training image and the category label data of that training image to obtain the first cross-entropy. This yields at least two first cross-entropies corresponding one-to-one with at least two training images. Similarly, cross-entropy can be calculated based on the fourth category probability distribution and the category label data of the mixed image to obtain the second cross-entropy. Then, a second sum of at least two first cross-entropies and the second cross-entropy can be calculated.

[0127] Assuming there are at least two training images, specifically two training images, and the category label data corresponding to these two training images are represented by y. i y j The probability distribution of the third category corresponding to each of the two training images is represented as P. i P j The category label data corresponding to the mixed image is represented as The fourth category probability distribution corresponding to the mixed image is represented as follows: The second sum is represented as L task Then we have:

[0128]

[0129] Among them, -y i logp i and -y j logp j Each represents a first cross-entropy. This represents the second cross-entropy.

[0130] Optionally, the second sum can be directly used as the model loss value of the neural network model; or, the second sum can be used as a component of the model loss value of the neural network model, and calculated in conjunction with the second sum and other components of the model loss value (such as L mentioned above). probability L feature After that, the model loss value of the neural network model is obtained by summing the results.

[0131] In this implementation, for each training image in at least two training images, cross-entropy can be calculated based on the third feature map corresponding to the training image and the category label data of the training image. Cross-entropy can also be calculated based on the second mixed feature map and the category label data of the mixed image. Combined with addition operations, the model loss value of the neural network model can be obtained efficiently and reliably. Furthermore, using the obtained model loss value for training the neural network model can make the image classification results obtained by the neural network model when performing image classification tasks as consistent as possible with the real situation, thereby improving the prediction accuracy of the neural network model.

[0132] In the embodiments of this disclosure, a classifier in the neural network model generates at least two third feature maps for at least two training images and a fourth feature map for the mixed image. A second mixed feature map of the at least two third feature maps is determined, and the model loss value of the neural network model is determined by comparing the fourth feature map with the second mixed feature map. This is equivalent to mining additional knowledge across images (i.e., mining additional knowledge based on at least two training images and the mixed image), and the mined knowledge is used for training the neural network model. This can capture the relationship dependencies between different images, improve the richness and diversity of knowledge used during model training, thereby improving the performance of the trained neural network model and further reducing training overhead.

[0133] exist Figure 1 Based on the illustrated embodiments, as Figure 5 As shown, step 130 includes steps 1311, 1315 and 1317.

[0134] Step 1311: Based on the processing results, perform loss calculations on the neural network model in at least two ways to obtain at least two calculated loss values.

[0135] It should be noted that the above text details L. probability L feature L task In practice, the calculation method described above can be used to obtain L. probability L feature L task At least two of them are used as at least two calculated loss values ​​in step 1311.

[0136] Step 1315: Determine at least two third sums of the calculated loss values.

[0137] Assuming at least two calculated loss values ​​specifically include L probability L feature L task The sum of the three is denoted as L, then we have:

[0138] L = L feature +L probability +L task

[0139] Step 1317: Based on the third sum, determine the model loss value of the neural network model.

[0140] Alternatively, the third sum can be directly determined as the model loss value; or, the third sum can be mapped to a specified interval (e.g., 0 to 1, 0 to 5, etc.) and the resulting mapped value can be used as the model loss value.

[0141] like Figure 2 As shown, in order to train a neural network model for image classification tasks, two different original images (equivalent to the training images mentioned above) can be randomly obtained first, and then a hybrid image can be constructed based on these two original images.

[0142] Next, through inference by the neural network model, we can obtain feature maps as the feature extraction result of the original image (equivalent to the first feature map in the case where the predetermined sub-network includes a feature extractor) and feature maps carrying the class probability distribution of the original image (equivalent to the first feature map in the case where the predetermined sub-network includes a classifier). We also obtain feature maps as the feature extraction result of the mixed image (equivalent to the second feature map in the case where the predetermined sub-network includes a feature extractor) and feature maps carrying the class probability distribution of the mixed image (equivalent to the second feature map in the case where the predetermined sub-network includes a classifier). Using the L2 loss function, we can make the feature map obtained by aggregating the feature maps used as the feature extraction result as similar as possible to the feature map generated by feature extraction from the mixed image, thereby enabling bidirectional knowledge transfer at the feature level (corresponding to...). Figure 2 The section marked "feature distillation from network knowledge" uses a relative entropy loss function to aggregate the feature maps carrying class probability distributions, making the resulting class probability distribution as similar as possible to the class probability distribution generated for the mixed image. This enables bidirectional knowledge transfer at the probabilistic level (corresponding to...). Figure 2 The section marked "probability distribution from network knowledge distillation" is included. Additionally, error calculation can be performed using the cross-entropy loss function based on the category label data of the original and mixed images (corresponding to...). Figure 2 (The part marked with "real label").

[0143] Using the method described above, a total of multiple loss values ​​can be obtained (such as L mentioned above). probability L feature L taskThe model loss value is obtained by summing multiple calculated loss values. The model loss value can effectively reflect the errors in the feature extractor of the neural network model, the errors in the classifier of the neural network model, and the overall error of the neural network model. This is conducive to obtaining a neural network model with superior image classification performance through end-to-end training, and the trained neural network model can be directly deployed.

[0144] In summary, the embodiments of this disclosure, by employing a model training method based on self-network knowledge distillation, can effectively improve the image classification performance of neural network models. Compared to knowledge distillation methods using a teacher-student training framework, this method eliminates the need for pre-training a teacher network. Instead, it uses at least two random training images and mixed images to guide the knowledge distillation of a single neural network model, thus simplifying the training process into an end-to-end process and significantly reducing training overhead. Furthermore, the embodiments of this disclosure perform additional knowledge mining across images, thereby capturing the dependencies between different images and enhancing the richness and diversity of knowledge utilized during model training. This further improves the performance of the neural network model while further reducing training overhead.

[0145] Any of the image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0146] Exemplary device

[0147] Figure 6 This is a schematic diagram of the structure of a model training apparatus based on self-network knowledge distillation provided in an exemplary embodiment of this disclosure. Figure 6 The apparatus shown includes a first determining module 610, a processing module 620, a second determining module 630, and a training module 640.

[0148] The first determining module 610 is used to determine a blended image of at least two training images;

[0149] The processing module 620 is used to process at least two training images and the mixed image determined by the first determining module 610 through a neural network model to obtain a processing result;

[0150] The second determining module 630 is used to determine the model loss value of the neural network model based on the processing result obtained by the processing module 620.

[0151] The training module 640 is used to train the neural network model based on the model loss value determined by the second determining module 630.

[0152] In an optional example, such as Figure 7 As shown, the processing module 620 includes:

[0153] The first generation submodule 6201 is used to generate at least two first feature maps for at least two training images through a predetermined subnetwork in the neural network model, and to generate a second feature map for the mixed image determined by the first determination module 610.

[0154] The first determining submodule 6203 is used to determine the processing result, which includes: at least two frames of first feature maps and second feature maps generated by the first generating submodule 6201.

[0155] The second determining module 630 includes:

[0156] The second determining submodule 6301 is used to determine the first mixed feature map of at least two frames of first feature maps generated by the first generating submodule 6201;

[0157] The third determining submodule 6303 is used to determine the model loss value of the neural network model by comparing the second feature map generated by the first generating submodule 6201 with the first mixed feature map determined by the second determining submodule 6301.

[0158] In one optional example, the predefined subnetwork includes: a classifier in a neural network model;

[0159] The third determining submodule 6303 includes:

[0160] The first determining unit is used to determine the probability distribution of the first category based on the second feature map generated by the first generating submodule 6201.

[0161] The second determining unit is used to determine the second category probability distribution based on the first mixed feature map determined by the second determining submodule 6301.

[0162] The first calculation unit is used to calculate the first relative entropy of the first category probability distribution determined by the first determining unit with respect to the second category probability distribution determined by the second determining unit;

[0163] The second calculation unit is used to calculate the second relative entropy of the second category probability distribution determined by the second determining unit with respect to the first category probability distribution determined by the first determining unit;

[0164] The third calculation unit is used to calculate the first sum of the first relative entropy calculated by the first calculation unit and the second relative entropy calculated by the second calculation unit;

[0165] The third determining unit is used to determine the model loss value of the neural network model based on the first sum calculated by the third calculation unit.

[0166] In one optional example, the predefined subnetwork includes: a feature extractor in a neural network model;

[0167] The third determining submodule 6303 includes:

[0168] The fourth calculation unit is used to calculate the average squared error between the second feature map generated by the first generation submodule 6201 and the first mixed feature map determined by the second determination submodule 6301.

[0169] The fourth determining unit is used to determine the model loss value of the neural network model based on the mean squared error value determined by the fourth calculation unit.

[0170] In an optional example, such as Figure 8 As shown, the processing module 620 includes:

[0171] The second generation submodule 6205 is used to generate at least two frames of third feature maps for at least two training images using a classifier in the neural network model, and to generate a fourth feature map for the mixed image determined by the first determination module 610.

[0172] The fourth determining submodule 6207 is used to determine the processing result, which includes: at least two frames of third feature maps and the fourth feature map generated by the second generation submodule 6205;

[0173] The second determining module 630 includes:

[0174] The fifth determining submodule 6305 is used to determine the second mixed feature map of at least two frames of third feature maps generated by the second generating submodule 6205;

[0175] The sixth determining submodule 6307 is used to determine the category label data of the mixed image determined by the first determining module 610 based on the category label data of at least two training images.

[0176] The seventh determining submodule 6309 is used to determine the model loss value of the neural network model based on at least two frames of third feature maps generated by the second generating submodule 6205, the second mixed feature map determined by the fifth determining submodule 6305, the class label data of each of the at least two frames of training images, and the class label data of the mixed image determined by the sixth determining submodule 6307.

[0177] In one optional example, the seventh determined submodule 6309 includes:

[0178] The fifth determining unit is used to determine the third category probability distribution for each training image in at least two training images, based on the third feature map corresponding to the training image generated by the second generation submodule 6205.

[0179] The sixth determining unit is used to determine the probability distribution of the fourth category based on the second mixed feature map determined by the fifth determining submodule 6305;

[0180] The fifth calculation unit is used to calculate the first cross-entropy for each training image in at least two training images, based on the third category probability distribution corresponding to the training image determined by the fifth determination unit and the category label data of the training image.

[0181] The sixth calculation unit is used to calculate the second cross-entropy based on the fourth category probability distribution determined by the sixth determining unit and the category label data of the mixed image determined by the sixth determining submodule 6307;

[0182] The seventh calculation unit is used to calculate the second sum of at least two first cross-entropies corresponding to at least two training images calculated by the fifth calculation unit and the second cross-entropy calculated by the sixth calculation unit;

[0183] The seventh determining unit is used to determine the model loss value of the neural network model based on the second sum.

[0184] In an optional example, such as Figure 9 As shown, the second determining module 630 includes:

[0185] The calculation submodule 6311 is used to perform loss calculation on the neural network model in at least two ways based on the processing result obtained by the processing module 620, and obtain at least two calculated loss values.

[0186] The eighth determining submodule 6313 is used to determine the third sum of at least two calculated loss values ​​obtained by the calculation submodule 6311;

[0187] The ninth determining submodule 6315 is used to determine the model loss value of the neural network model based on the third sum determined by the eighth determining submodule 6313.

[0188] Exemplary electronic devices

[0189] Below, for reference Figure 10 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0190] Figure 10A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0191] like Figure 10 As shown, the electronic device 1000 includes one or more processors 1010 and memory 1020.

[0192] The processor 1010 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0193] The memory 1020 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium. The processor 1010 may execute the program instructions to implement the model training method based on self-network knowledge distillation of the various embodiments of this disclosure described above. Additionally, the processor 1010 may also execute the program instructions to implement other desired functions.

[0194] In one example, the electronic device 1000 may also include an input device 1030 and an output device 1040, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0195] For example, when the electronic device 1000 is a first device or a second device, the input device 1030 may be a microphone or a microphone array. When the electronic device 1000 is a standalone device, the input device 1030 may be a communication network connector for receiving acquired input signals from the first device and the second device.

[0196] In addition, the input device 1030 may also include, for example, a keyboard, a mouse, etc. The output device 1040 can output various information to the outside. The output device 1040 may include, for example, a monitor, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0197] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 1000 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1000 may include any other suitable components depending on the specific application.

[0198] Exemplary computer program products and computer-readable storage media

[0199] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the model training method based on self-network knowledge distillation according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0200] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0201] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the model training method based on self-network knowledge distillation according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0202] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0204] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0205] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0206] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0207] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0208] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0209] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A model training method based on self-network knowledge distillation, comprising: determining a mixed image of at least two training images; processing the at least two training images and the mixed image by a neural network model to obtain a processing result; determining a model loss value of the neural network model based on the processing result; training the neural network model based on the model loss value; wherein the processing the at least two training images and the mixed image by the neural network model to obtain the processing result comprises: generating, by a predetermined sub-network in the neural network model, at least two first feature maps for the at least two training images and a second feature map for the mixed image, the processing result comprising the at least two first feature maps and the second feature map; the determining the model loss value of the neural network model based on the processing result comprises: determining a first mixed feature map of the at least two first feature maps; and determining the model loss value of the neural network model by comparing the second feature map and the first mixed feature map; or, wherein the processing the at least two training images and the mixed image by the neural network model to obtain the processing result comprises: generating, by a classifier in the neural network model, at least two third feature maps for the at least two training images and a fourth feature map for the mixed image, the processing result comprising the at least two third feature maps and the fourth feature map; the determining the model loss value of the neural network model based on the processing result comprises: determining a second mixed feature map of the at least two third feature maps; determining a class label data of the mixed image based on class label data of the at least two training images respectively; and determining the model loss value of the neural network model based on the at least two third feature maps, the second mixed feature map, the class label data of the at least two training images respectively, and the class label data of the mixed image.

2. The method of claim 1, wherein, the predetermined sub-network comprises a classifier in the neural network model; the determining the model loss value of the neural network model by comparing the second feature map and the first mixed feature map comprises: determining a first class probability distribution based on the second feature map; determining a second class probability distribution based on the first mixed feature map; calculating a first relative entropy of the first class probability distribution with respect to the second class probability distribution; calculating a second relative entropy of the second class probability distribution with respect to the first class probability distribution; calculating a first sum of the first relative entropy and the second relative entropy; determining the model loss value of the neural network model based on the first sum.

3. The method of claim 1, wherein, the predetermined sub-network comprises a feature extractor in the neural network model; the determining the model loss value of the neural network model by comparing the second feature map and the first mixed feature map comprises: calculating a mean squared error value between the second feature map and the first mixed feature map; and Determine a model loss value of the neural network model based on the average square error value.

4. The method of claim 1, wherein, The determination of the model loss value of the neural network model based on the at least two third feature maps, the second mixed feature map, the class label data of the at least two training images respectively, and the class label data of the mixed image comprises: For each of the at least two training images, determine a third class probability distribution based on the third feature map corresponding to the training image; Determine a fourth class probability distribution based on the second mixed feature map; For each of the at least two training images, calculate a first cross-entropy based on the third class probability distribution corresponding to the training image and the class label data of the training image; Calculate a second cross-entropy based on the fourth class probability distribution and the class label data of the mixed image; Calculate a second sum of the at least two first cross-entropies corresponding to the at least two training images and the second cross-entropy; Determine the model loss value of the neural network model based on the second sum.

5. The method of any one of claims 1-4, wherein, The determination of the model loss value of the neural network model based on the processing result comprises: Based on the processing result, perform loss calculation on the neural network model in at least two ways to obtain at least two calculated loss values; Determine a third sum of the at least two calculated loss values; Determine the model loss value of the neural network model based on the third sum.

6. A model training device based on self-network knowledge distillation, comprising: A first determination module for determining a mixed image of at least two training images; A processing module for processing the at least two training images and the mixed image determined by the first determination module through a neural network model to obtain a processing result; A second determination module for determining a model loss value of the neural network model based on the processing result obtained by the processing module; A training module for training the neural network model based on the model loss value determined by the second determination module; The processing module comprises: a first generation submodule for generating at least two first feature maps for at least two training images and a second feature map for the mixed image through a predetermined subnetwork in the neural network model; and a first determination submodule for determining a processing result, the processing result comprising the at least two first feature maps and the second feature map; The second determination module comprises: a second determination submodule for determining a first mixed feature map of the at least two first feature maps generated by the first generation submodule; and a third determination submodule for determining a model loss value of the neural network model by comparing the second feature map with the first mixed feature map; Or, The processing module comprises: a second generation submodule for generating at least two third feature maps for at least two training images and a fourth feature map for the mixed image through a classifier in the neural network model; and a fourth determination submodule for determining a processing result, the processing result comprising the at least two third feature maps and the fourth feature map; The second determining module comprises: a fifth determining submodule, configured to determine a second mixed feature map of the at least two frames of third feature maps generated by the second generating submodule; a sixth determining submodule, configured to determine a class label data of the mixed image based on the class label data of each of the at least two frames of training images; and a seventh determining submodule, configured to determine a model loss value of the neural network model based on the at least two frames of third feature maps, the second mixed feature map, the class label data of each of the at least two frames of training images, and the class label data of the mixed image. 7.A computer readable storage medium, the storage medium storing a computer program, the computer program being configured to execute the model training method based on self-network knowledge distillation according to any one of claims 1-5. 8.An electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the model training method based on self-network knowledge distillation according to any one of claims 1-5.

Images