Knowledge distillation method and device of image data, terminal equipment and storage medium

Abstract

This application relates to the field of knowledge distillation technology, and provides a method, apparatus, terminal device, and storage medium for knowledge distillation of image data. By employing an attention-based feature distillation method to guide the training of a student network using image data and a total loss function, error constraints are applied to the attention maps of the low- and mid-level output feature maps of the teacher and student networks. A similarity-based feature constraint method is used to apply similarity constraints to the feature vectors of the high-level output feature maps of the teacher and student networks. The total loss function is updated based on the attention map loss function, the similarity loss function, and the classification loss function of the student network's classification layer. Using image data and the updated loss function, the teacher network guides the student network through iterative training, ensuring that the accuracy, speed, and complexity of the student network after knowledge distillation meet the performance requirements of image classification tasks.

Description

Technical Field

[0001] This application belongs to the field of knowledge distillation technology, and in particular relates to a knowledge distillation method, apparatus, terminal equipment and storage medium for image data. Background Technology

[0002] Knowledge distillation is a method that extracts knowledge from a complex, large model (teacher network) and compresses it into a single, smaller model (student network). The larger model then guides the smaller model to improve its performance. The output of the teacher network is used as the soft target for training the student network. In practical applications, due to computational limitations, smaller models are often needed to perform image classification tasks. However, the performance of smaller models is significantly lower than that of larger models, thus requiring knowledge distillation for model compression.

[0003] When using traditional knowledge distillation methods for image classification, knowledge transfer is achieved using the probability distribution of the final classification layer. However, the training data for the classification layer can reach hundreds of thousands or even tens of millions of records, making direct knowledge distillation using the classification layer ineffective. Furthermore, traditional knowledge distillation methods typically employ similar network structures for the teacher and student networks, while the network structures of the large and small models used for image classification differ significantly. Traditional knowledge distillation methods perform poorly on such heterogeneous network structures. Summary of the Invention

[0004] In view of this, embodiments of this application provide a knowledge distillation method, apparatus, terminal device, and storage medium for image data to solve the problem of poor performance when using traditional knowledge distillation methods for image classification.

[0005] A first aspect of this application provides a knowledge distillation method for image data, comprising:

[0006] Image data is input into the trained teacher network and the student network to be trained;

[0007] Based on the total loss function, the teacher network is used to guide the training of the student network.

[0008] Obtain the attention map of the feature map output at the i-th level of the teacher network and the attention map of the feature map output at the i-th level of the student network;

[0009] Error constraints are applied to the attention maps of the feature maps output at the i-th level of the teacher network and the attention maps of the feature maps output at the i-th level of the student network to obtain the loss function of the i-th attention map.

[0010] A similarity constraint is applied to the feature map output at the m-th level of the teacher network and the feature map output at the m-th level of the student network to obtain a similarity loss function;

[0011] Based on the classification loss function of the classification layer of the student network, the i-th attention map loss function, and the similarity loss function, update the total loss function, and return to execute the step of guiding the training of the student network using the teacher network based on the total loss function;

[0012] Where i = 1, 2, ..., m-1, and m is an integer greater than 1.

[0013] A second aspect of this application provides a knowledge distillation apparatus for image data, comprising:

[0014] An image data input unit is used to input image data into the trained teacher network and the student network to be trained;

[0015] A knowledge distillation unit is used to guide the training of the student network using the teacher network based on the total loss function;

[0016] Attention map acquisition unit, used to acquire the attention map of the feature map output of the i-th level of the teacher network and the attention map of the feature map output of the i-th level of the student network;

[0017] The error constraint unit is used to apply error constraints to the attention map of the feature map output by the i-th level of the teacher network and the attention map of the feature map output by the i-th level of the student network, so as to obtain the loss function of the i-th attention map.

[0018] The similarity constraint unit is used to apply similarity constraints to the feature map output at the m-th level of the teacher network and the feature map output at the m-th level of the student network, thereby obtaining a similarity loss function.

[0019] The total loss function update unit is used to update the total loss function according to the classification loss function of the classification layer of the student network, the i-th attention map loss function, and the similarity loss function, and return to execute the step of guiding the training of the student network using the teacher network based on the total loss function;

[0020] Where i = 1, 2, ..., m-1, and m is an integer greater than 1.

[0021] A third aspect of this application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the knowledge distillation method for image data as provided in the first aspect of this application.

[0022] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the knowledge distillation method for image data as provided in the first aspect of this application.

[0023] The knowledge distillation method for image data provided in the first aspect of this application involves employing an attention-based feature distillation approach during the process of using image data and a total loss function to guide the training of a student network by a teacher network. This involves applying error constraints to the attention maps of the low- and mid-level output feature maps of the teacher and student networks to obtain an attention map loss function. Then, a similarity-based feature constraint approach is used to apply similarity constraints to the feature vectors of the high-level output feature maps of the teacher and student networks to obtain a similarity loss function. Finally, the total loss function is updated based on the attention map loss function, the similarity loss function, and the classification loss function of the student network's classification layer. Using the image data and the updated total loss function, the teacher network guides the student network through iterative training, ensuring that the accuracy, speed, and complexity of the student network after knowledge distillation meet the performance requirements of the image classification task.

[0024] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic flowchart of the first method for knowledge distillation of image data provided in the embodiments of this application;

[0027] Figure 2 This is a schematic diagram of the second process of the knowledge distillation method for image data provided in the embodiments of this application;

[0028] Figure 3 This is a schematic diagram of the third process of the knowledge distillation method for image data provided in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram of the fourth process of the knowledge distillation method for image data provided in the embodiments of this application;

[0030] Figure 5This is a schematic diagram of the structure of the teacher network and student network provided in the embodiments of this application;

[0031] Figure 6 This is a schematic diagram of the knowledge distillation apparatus provided in the embodiments of this application;

[0032] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0033] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0034] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0035] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0036] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0037] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0038] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0039] The knowledge distillation method for image data provided in this application embodiment can be executed by the processor of a terminal device when running a computer program with corresponding functions. By utilizing a feature distillation method based on attention of feature maps at multiple stages, attention maps of the low- and mid-level output feature maps of the teacher network and the student network are obtained. This enables the low- and mid-level features of the student network to better fit the low- and mid-level feature extraction capabilities of the teacher network. By performing similarity-based feature distillation on the high-level feature maps, the high-level output feature maps of the student network are made to fit the high-level output feature maps of the teacher network as closely as possible, so that the student network obtains a feature expression capability close to that of the teacher network.

[0040] In this application, image data is a training dataset composed of multiple images. Image data can include any number of images that need to be classified or recognized, and can include at least one of the following: facial images, industrial product images, animal images, landscape images, etc. The terminal device can be a computing device with at least one function such as image classification, image recognition, or machine vision, for example, a robot, mobile phone, personal digital assistant (PDA), self-service terminal, camera, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, tablet computer, laptop computer, personal computer (PC), netbook, server, access control system, security equipment, etc. This application embodiment does not impose any restrictions on the specific type of terminal device.

[0041] like Figure 1 As shown, the knowledge distillation method for image data provided in this application includes the following steps S101 to S106:

[0042] Step S101: Input the image data into the trained teacher network and the student network to be trained, and proceed to step S102.

[0043] In applications, image data can be obtained from the local storage space of the terminal device, or from the storage space of any other device communicating with the terminal device (e.g., a server), for example, from multiple images obtained from an open image library.

[0044] In applications, the teacher network is a pre-trained network capable of meeting the classification or recognition requirements of image data; that is, the classification loss function of the classification layer converges or meets the accuracy requirements. The teacher network can employ heavyweight convolutional neural networks (CNNs), such as Residual Networks (ResNet), Recurrent Neural Networks (RNNs), and Dense Convolutional Networks (DenseNet). ResNet can be ResNet 50, ResNet 101, or ResNet 152.

[0045] In applications, the student network is a training network that needs to utilize knowledge distillation methods for knowledge transfer learning, learning image data classification or recognition functions under the guidance of the teacher network. The student network can employ lightweight convolutional neural networks, such as MobileNet, ShuffleNet, or ThunderNet. MobileNet can be MobileNetv1, MobileNetv2, or MobileNetv3. When the terminal device uses it for face image classification or face recognition, MobileNet can be the mobile face recognition network MobileFaceNet. ShuffleNet can be ShuffleNetv1 or ShuffleNetv2.

[0046] Step S102: Based on the total loss function, use the teacher network to guide the student network training, and proceed to step S103.

[0047] In the application, the teacher network guides the student network through iterative training. The training objective is for the classification loss function of the student network's classification layer to converge or meet accuracy requirements. In the first training epoch or the first training phase, the loss function can consist solely of the classification loss function of the student network's classification layer. The first training phase can include multiple training epochs. The classification loss function of the student network's classification layer can be set to be the same as that of the teacher network's classification layer. However, since the student network achieves performance as close as possible to the teacher network through transfer learning, it may not necessarily reach the same performance. Therefore, the classification loss function of the student network's classification layer can also be set differently from the teacher network's classification loss function, being slightly larger. The classification loss function can be a normalized exponential function (Softmax Faction). When the terminal device is used for face image classification or face recognition, the classification loss function can be the CosFace loss function, ArcFace loss function, etc.

[0048] Step S103: Obtain the attention map of the feature map output at the i-th level of the teacher network and the attention map of the feature map output at the i-th level of the student network, and proceed to step S104.

[0049] In applications, the stacked convolutional layers of the teacher and student networks can be pre-divided into at least m stages, each stage including at least one convolutional layer. These stages belong to the backbone network. The backbone network is used to extract features from the image, obtaining feature maps with high discriminative power and robustness. These extracted feature maps serve as input to the subsequent detection head network. The backbone network can be constructed based on ordinary convolutional layers, depthwise separable convolutional layers, depthwise separable dilated convolutional layers, batch normalization layers, Mish activation function layers, average pooling layers, etc.

[0050] In applications, the specific method for obtaining attention maps is as follows:

[0051] Obtain the attention maps of the feature maps of the first-level output of the teacher network and the first-level output of the student network;

[0052] Obtain the attention maps of the feature maps of the second-level output of the teacher network and the second-level output of the student network;

[0053] ...;

[0054] This process continues until the attention maps of the feature maps output at the (m-1)th level of the teacher network and the student network are obtained. That is, let i = 1, 2, ..., m-1, where m is an integer greater than 1.

[0055] In applications, the value of m can be set according to actual needs; for example, m = 4.

[0056] In one embodiment, the feature maps output at the low and mid levels of the teacher network and the student network are local feature maps, and the feature maps output at the high levels of the teacher network and the student network are global feature maps.

[0057] In applications, the low and mid-levels of the student network (e.g., levels 1 to m-1) are used to learn local features of the image, while the high-level layers (e.g., level m) are used to learn global features of the image. When the terminal device is used for face image classification or face recognition, local features can be facial key point information (e.g., corner points, facial features (the centers of the left and right eyes, the nose, and the left and right corners of the mouth)), while global features can be comprehensive features of the entire face.

[0058] In applications, feature maps can include multiple dimensions, such as batch, channel, and spatial dimensions. The batch dimension reflects the batch size of the feature map, i.e., the number of feature maps output at a given level. The channel dimension reflects the number of channels in the feature map, i.e., the number of color channels and the bit depth of each color channel. For example, a feature map might contain three color channels: Red (R), Green (G), and Blue (B), each with 8 bits. The spatial dimension reflects the spatial size of the feature map, such as height and width. When a feature map includes these three dimensions, the dimension representation can be (C, H, W) or (B, C, H, W); where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width.

[0059] In applications, attention maps of the feature map can be obtained in different dimensional directions based on the feature map's dimensions, such as at least one of the channel dimension attention map in the channel dimension direction and the spatial dimension attention map in the spatial dimension direction.

[0060] like Figure 2As shown, in one embodiment, step S103 includes the following steps S201 and S202:

[0061] Step S201: In the channel dimension direction, pool the feature map output by the i-th layer of the teacher network to obtain the channel dimension attention map of the feature map output by the i-th layer of the teacher network.

[0062] Step S202: In the channel dimension direction, pool the feature map output of the i-th layer of the student network to obtain the channel dimension attention map of the feature map output of the i-th layer of the student network.

[0063] In the application, pooling is performed on the feature maps output by the i-th level of the teacher network and the i-th level of the student network in the channel dimension, respectively, to obtain the channel dimension attention maps of the feature maps output by the i-th level of the teacher network and the i-th level of the student network. When the dimensional representation format of the feature map is (C,H,W), the dimensional representation format of the channel dimension attention map is (C,1,1); when the dimensional representation format of the feature map is (B,C,H,W), the dimensional representation format of the channel dimension attention map is (B,C,1,1).

[0064] like Figure 3 As shown, in one embodiment, step S103 includes the following steps S301 and S302:

[0065] Step S301: In the spatial dimension, pool the feature map output by the i-th layer of the teacher network to obtain the spatial dimension attention map of the feature map output by the i-th layer of the teacher network.

[0066] Step S302: In the spatial dimension, pool the feature map output of the i-th layer of the student network to obtain the spatial dimension attention map of the feature map output of the i-th layer of the student network.

[0067] In the application, pooling is performed on the feature maps output by the i-th level of the teacher network and the i-th level of the student network in the spatial dimension, respectively, to obtain the spatial dimension attention maps of the feature maps output by the i-th level of the teacher network and the student network. When the dimensional representation of the feature map is (C, H, W), the dimensional representation of the spatial dimension attention map is (1, H, W); when the dimensional representation of the feature map is (B, C, H, W), the dimensional representation of the spatial dimension attention map is (B, 1, H, W).

[0068] In applications, pooling can include at least one of max pooling and mean pooling (or average pooling). Pooling downsamples the feature map to reduce its size, thereby improving the processing speed of teacher and student networks when classifying or recognizing image data. Pooling along the channel dimension reduces the number of channels, while pooling along the spatial dimension reduces the spatial size.

[0069] Step S104: Apply error constraints to the attention map of the feature map output at the i-th level of the teacher network and the attention map of the feature map output at the i-th level of the student network to obtain the loss function of the i-th attention map, and proceed to step S105.

[0070] In the application, the error between the attention maps of the feature maps output at the same level of the teacher network and the student network is calculated sequentially, and the error of each level is constrained by the corresponding error constraints to obtain the attention map loss function corresponding to each level.

[0071] In application, the specific method for obtaining the attention map loss function is as follows:

[0072] Error constraints are applied to the attention maps of the feature maps output by the first level of the teacher network and the first level of the student network to obtain the first attention map loss function;

[0073] Error constraints are applied to the attention maps of the feature maps output by the second level of the teacher network and the second level of the student network to obtain the loss function of the second attention map.

[0074] ...;

[0075] This process continues until error constraints are applied to the attention maps of the feature maps output at the (m-1)th level of the teacher network and the (m-1)th level of the student network, resulting in the (m-1)th attention map loss function.

[0076] like Figure 2 As shown, in one embodiment, step S104 includes:

[0077] Step S401: Apply error constraints to the channel dimension attention map of the feature map output by the i-th level of the teacher network and the channel dimension attention map of the feature map output by the i-th level of the student network to obtain the loss function of the i-th channel dimension attention map.

[0078] In the application, after obtaining the channel dimension attention maps of the feature maps output by each level of the teacher network and the feature maps output by each level of the student network in the channel dimension direction, the channel dimension attention maps of the feature maps output by the corresponding level of the teacher network and the feature maps output by the corresponding level of the student network are further obtained and error constraints are applied to obtain the channel dimension attention map loss function corresponding to the corresponding level.

[0079] like Figure 3 As shown, in one embodiment, step S104 includes:

[0080] Step S402: Apply error constraints to the spatial dimension attention map of the feature map output by the i-th level of the teacher network and the spatial dimension attention map of the feature map output by the i-th level of the student network to obtain the i-th spatial dimension attention map loss function.

[0081] In the application, after obtaining the spatial dimension attention maps of the feature maps of each level of the teacher network and the feature maps of each level of the student network in the spatial dimension, the spatial dimension attention maps of the feature maps of the corresponding level of the teacher network and the student network are further obtained to perform error constraints, and the spatial dimension attention map loss function corresponding to the corresponding level is obtained.

[0082] In applications, errors may include, but are not limited to, at least one of the following: mean square error (MSE), random error, and gross conceptual error (GSE).

[0083] Step S105: Apply similarity constraints to the feature map output at the m-th level of the teacher network and the feature map output at the m-th level of the student network to obtain the similarity loss function, and proceed to step S106.

[0084] In the application, after sequentially completing the error constraint between the attention maps of the feature maps of the first 1 to the (m-1)th levels of the teacher network and the student network, the similarity between the feature map of the m-th level output of the teacher network and the feature map of the m-th level output of the student network is further calculated, and the similarity is constrained by the corresponding similarity constraint conditions to obtain the corresponding similarity loss function.

[0085] In applications, similarity is used to reflect the closeness between the feature map output at the m-th level of the teacher network and the feature map output at the m-th level of the student network. The specific similarity may include, but is not limited to, at least one of the following: cosine similarity, Euclidean distance, Hamming distance, and Mahalanobis distance.

[0086] like Figure 4 As shown, in one embodiment, step S105 includes the following steps S501 to S504:

[0087] Step S501: Obtain the feature vector of the feature map output by the m-th layer of the teacher network and normalize it to obtain the first normalization result, then proceed to step S503.

[0088] Step S502: Obtain the feature vector of the feature map output by the m-th layer of the student network and normalize it to obtain the second normalization result, then proceed to step S503.

[0089] Step S503: Based on the first normalization result and the second normalization result, obtain the cosine value of the angle between the feature vector of the feature map output at the m-th level of the teacher network and the feature vector of the feature map output at the m-th level of the student network, and proceed to step S504.

[0090] Step S504: Constrain the magnitude of the cosine value based on the preset similarity constraint conditions to obtain the cosine similarity loss function.

[0091] In application, when the similarity is cosine similarity, the method for calculating the cosine similarity loss function is as follows: First, obtain the feature vectors of the feature maps output at the m-th level of the teacher network and the student network respectively, and normalize them to obtain the first normalization result corresponding to the teacher network and the second normalization result corresponding to the student network; then, multiply the first normalization result and the second normalization result to obtain the cosine value of the angle between the feature vectors of the feature maps output at the m-th level of the teacher network and the student network; finally, use the pre-set cosine similarity constraint condition to constrain the cosine value to obtain the cosine similarity loss function. The cosine similarity constraint condition is set based on the goal of making the angle between the feature vectors of the feature maps output at the m-th level of the teacher network and the student network as close to 0 as possible, that is, the cosine value as close to 1 as possible. Therefore, the cosine similarity condition can be set to make the cosine value infinitely close to 1.

[0092] In one embodiment, the cosine similarity loss function is calculated using the following formula:

[0093]

[0094] Among them, L m () represents the cosine similarity loss function, and d() represents the distance metric function. This represents the feature vector transformation function corresponding to the feature vector of the feature map output from the m-th layer of the teacher network. The feature vector represents the feature map output by the m-th layer of the teacher network. Let represent the feature vector transformation function corresponding to the feature vector of the feature map output at the m-th level of the student network. The feature vector represents the feature map output of the m-th layer of the student network.

[0095] In applications, the feature vector transformation function is used to normalize the feature vectors. A function with normalization function can be used according to actual needs, such as 1×1 convolution.

[0096] Step S106: Update the total loss function according to the classification loss function of the classification layer of the student network, the i-th attention map loss function, and the similarity loss function, and return to execute step S102.

[0097] In application, after obtaining the attention map loss functions corresponding to levels 1 to m-1 and the similarity loss function corresponding to level m, a new loss function is calculated based on the classification loss function of the student network's classification layer, the attention map loss functions corresponding to levels 1 to m-1, and the similarity loss function corresponding to level m. The loss function used in the current training cycle or the current training phase is then updated to the new loss function, so that the updated loss function can be used to train the student network in the next training cycle or the next training phase. The new loss function can be the sum of the classification loss function of the student network's classification layer, the attention map loss functions corresponding to levels 1 to m-1, and the similarity loss function corresponding to level m.

[0098] In one embodiment, the updated total loss function is calculated as follows:

[0099] L total =L cis +L attention +L similarity

[0100] Among them, L total Let L represent the total loss function. cis Let L represent the classification loss function. attention L represents the sum of the attention map loss functions from the 1st to the (m-1)th. similarity This represents the similarity loss function.

[0101] like Figure 5 As shown, an exemplary structural diagram of a teacher network and a student network is presented; where the left side is the teacher network and the right side is the student network, and the numbers 1 to m represent the first to the mth levels, respectively.

[0102] In one embodiment, after step S106, the following is included:

[0103] When the total loss function converges, the training of the student network is considered complete.

[0104] The image data to be classified or recognized is input into the trained student network, and the image recognition or classification results output by the student network are obtained.

[0105] This application also provides a knowledge distillation apparatus for image data, used to execute the steps in the above-described knowledge distillation method embodiments for image data. The knowledge distillation apparatus for image data can be a virtual appliance in a terminal device, run by the processor of the terminal device, or it can be the terminal device itself.

[0106] like Figure 6 As shown, the knowledge distillation apparatus 100 for image data provided in this application embodiment includes:

[0107] Image data input unit 101 is used to input image data into the trained teacher network and the student network to be trained;

[0108] Knowledge distillation unit 102 is used to guide the training of the student network using the teacher network based on the total loss function;

[0109] Attention map acquisition unit 103 is used to acquire the attention map of the feature map output of the i-th level of the teacher network and the attention map of the feature map output of the i-th level of the student network.

[0110] Error constraint unit 104 is used to apply error constraints to the attention map of the feature map output by the i-th level of the teacher network and the attention map of the feature map output by the i-th level of the student network to obtain the i-th attention map loss function.

[0111] The similarity constraint unit 105 is used to constrain the similarity between the feature map output at the m-th level of the teacher network and the feature map output at the m-th level of the student network to obtain a similarity loss function.

[0112] The total loss function update unit 106 is used to update the total loss function according to the classification loss function of the classification layer of the student network, the i-th attention map loss function and the similarity loss function, and return it to the image data input unit 101;

[0113] Where i = 1, 2, ..., m-1, and m is an integer greater than 1.

[0114] In one embodiment, the knowledge distillation apparatus further includes:

[0115] A determining unit is configured to determine that the student network training is complete when the total loss function converges;

[0116] An image processing unit is used to input image data to be processed into the trained student network and obtain the image processing result output by the student network. The image data to be processed includes at least one image to be classified or recognized.

[0117] In applications, the modules in the knowledge distillation device for image data can be software program modules, or they can be implemented through different logic circuits integrated in the processor, or they can be implemented through multiple distributed processors.

[0118] like Figure 7 As shown, this application embodiment also provides a terminal device 200, including: at least one processor 201 ( Figure 7 The diagram shows only one processor, memory 202, and computer program 203 stored in memory 202 and executable on at least one processor 201. When processor 201 executes computer program 203, it implements the steps in the various method embodiments described above.

[0119] In applications, terminal devices may include, but are not limited to, processors and memory. Those skilled in the art will understand that... Figure 7 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0120] In applications, the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0121] In applications, the memory may be an internal storage unit of the terminal device in some embodiments, such as the hard drive or RAM of the terminal device. In other embodiments, the memory may be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory may include both internal and external storage units of the terminal device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of a computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0122] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0124] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in the above-described method embodiments.

[0125] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0126] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0130] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A knowledge distillation method for image data, characterized in that, include: Image data is input into the trained teacher network and the student network to be trained; Based on the total loss function, the teacher network is used to guide the training of the student network. Obtain the attention map of the feature map output at the i-th level of the teacher network and the attention map of the feature map output at the i-th level of the student network; Error constraints are applied to the attention maps of the feature maps output at the i-th level of the teacher network and the attention maps of the feature maps output at the i-th level of the student network to obtain the loss function of the i-th attention map. A similarity constraint is applied to the feature map output at the m-th level of the teacher network and the feature map output at the m-th level of the student network to obtain a similarity loss function; Based on the classification loss function of the classification layer of the student network, the i-th attention map loss function, and the similarity loss function, update the total loss function, and return to execute the step of guiding the training of the student network using the teacher network based on the total loss function; Where i = 1, 2, ..., m-1, m is an integer greater than 1. In the first training cycle or the first stage of training, the total loss function is the classification loss function of the classification layer of the student network. The updated total loss function is the sum of the classification loss function of the classification layer of the student network, the i-th attention map loss function, and the similarity loss function.

2. The knowledge distillation method for image data as described in claim 1, characterized in that, The process of obtaining the attention map of the feature map output at the i-th level of the teacher network and the attention map of the feature map output at the i-th level of the student network includes: In the channel dimension direction, the feature map output by the i-th layer of the teacher network is pooled to obtain the channel dimension attention map of the feature map output by the i-th layer of the teacher network. In the channel dimension direction, the feature map output of the i-th layer of the student network is pooled to obtain the channel dimension attention map of the feature map output of the i-th layer of the student network. The step of applying error constraints to the attention maps of the feature maps output from the i-th level of the teacher network and the feature maps output from the i-th level of the student network to obtain the i-th attention map loss function includes: Error constraints are applied to the channel dimension attention map of the feature map output at the i-th level of the teacher network and the channel dimension attention map of the feature map output at the i-th level of the student network to obtain the loss function of the i-th channel dimension attention map.

3. The knowledge distillation method for image data as described in claim 1, characterized in that, The process of obtaining the attention map of the feature map output at the i-th level of the teacher network and the attention map of the feature map output at the i-th level of the student network includes: In the spatial dimension, the feature map output by the i-th layer of the teacher network is pooled to obtain the spatial dimension attention map of the feature map output by the i-th layer of the teacher network. In the spatial dimension, the feature map output by the i-th layer of the student network is pooled to obtain the spatial dimension attention map of the feature map output by the i-th layer of the student network. The step of applying error constraints to the attention maps of the feature maps output from the i-th level of the teacher network and the feature maps output from the i-th level of the student network to obtain the i-th attention map loss function includes: Error constraints are applied to the spatial dimension attention map of the feature map output at the i-th level of the teacher network and the spatial dimension attention map of the feature map output at the i-th level of the student network to obtain the i-th spatial dimension attention map loss function.

4. The knowledge distillation method for image data as described in claim 1, characterized in that, The similarity constraint is applied to the feature map output at the m-th level of the teacher network and the feature map output at the m-th level of the student network to obtain a similarity loss function, including: Obtain the feature vector of the feature map output by the m-th layer of the teacher network and normalize it to obtain the first normalization result; Obtain the feature vector of the feature map output at the m-th level of the student network and normalize it to obtain the second normalization result; Based on the first normalization result and the second normalization result, obtain the cosine value of the angle between the feature vector of the feature map output at the m-th level of the teacher network and the feature vector of the feature map output at the m-th level of the student network. The magnitude of the cosine value is constrained based on a preset cosine similarity constraint, resulting in a cosine similarity loss function.

5. The knowledge distillation method for image data as described in any one of claims 1 to 4, characterized in that, The error constraints include mean square error constraints, and the similarity constraints include cosine similarity constraints.

6. The knowledge distillation method for image data as described in any one of claims 1 to 4, characterized in that, The feature maps output at the low and mid levels of the teacher network and the student network are local feature maps, while the feature maps output at the high levels of the teacher network and the student network are global feature maps.

7. The knowledge distillation method for image data as described in any one of claims 1 to 4, characterized in that, The image data contains multiple facial images.

8. A knowledge distillation apparatus for image data, characterized in that, include: An image data input unit is used to input image data into the trained teacher network and the student network to be trained; A knowledge distillation unit is used to guide the training of the student network using the teacher network based on the total loss function; Attention map acquisition unit, used to acquire the attention map of the feature map output of the i-th level of the teacher network and the attention map of the feature map output of the i-th level of the student network; The error constraint unit is used to apply error constraints to the attention map of the feature map output by the i-th level of the teacher network and the attention map of the feature map output by the i-th level of the student network, so as to obtain the loss function of the i-th attention map. The similarity constraint unit is used to apply similarity constraints to the feature map output at the m-th level of the teacher network and the feature map output at the m-th level of the student network, thereby obtaining a similarity loss function. The total loss function update unit is used to update the total loss function according to the classification loss function of the classification layer of the student network, the i-th attention map loss function, and the similarity loss function, and return to execute the step of guiding the training of the student network using the teacher network based on the total loss function; Where i = 1, 2, ..., m-1, m is an integer greater than 1. In the first training cycle or the first stage of training, the total loss function is the classification loss function of the classification layer of the student network. The updated total loss function is the sum of the classification loss function of the classification layer of the student network, the i-th attention map loss function, and the similarity loss function.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the knowledge distillation method for image data as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the knowledge distillation method for image data as described in any one of claims 1 to 7.

Images