Knowledge distillation method and related apparatus

Abstract

Provided in the present application are a knowledge distillation method and a related apparatus. The method is applied in the field of image processing. The knowledge distillation method comprises: acquiring a training image, inputting the training image into a trained teacher network, and extracting a feature outputted by a kth intermediate layer in the teacher network; inputting the training image into a student network, and extracting a feature outputted by an sth intermediate layer in the student network, wherein s and k are the same or different; respectively mapping to a latent space the feature outputted by the kth intermediate layer and the feature outputted by the sth intermediate layer, so as to respectively obtain a first feature and a second feature; and performing combination processing on the first feature and the second feature in the latent space, so as to obtain a feature which has been subjected to the combination processing, mapping the feature which has been subjected to the combination processing back to the space in which the feature outputted by the intermediate layer in the teacher network is located, so as to obtain a mixed feature, and on the basis of the mixed feature, training the student network. The knowledge distillation method provided in the present application has a relatively good application effect in a super-resolution application scenario.

Description

A knowledge distillation method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202410224957.8, filed on February 28, 2024, entitled "A Method and Apparatus for Knowledge Distillation", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and more particularly to a knowledge distillation method and related apparatus. Background Technology

[0003] Knowledge distillation (KD) is a deep learning-based model compression technique that aims to train a small, simplified model (usually called a student model or student network) by extracting knowledge from a large, complex model (often called the teacher model or teacher network). The core idea of ​​knowledge distillation is to transfer knowledge from the teacher model to the student model, allowing the student model to retain as much of the teacher model's knowledge and performance as possible while also being lightweight and efficient. Knowledge distillation compression methods have wide applications in fields such as natural language processing and image processing.

[0004] Currently, common knowledge distillation methods, such as feature-affinity based knowledge distillation for efficient image super-resolution (FAKD) and contrastive self-distillation (CSD), do not perform well in some super-resolution (SR) scenarios due to the inherent characteristics of knowledge distillation algorithms. Summary of the Invention

[0005] This application provides a knowledge distillation method and related apparatus, which can be applied to various super-resolution scenarios and achieve good results in different super-resolution scenarios.

[0006] In a first aspect, this application provides a knowledge distillation method, comprising: acquiring training images; inputting the training images into a trained teacher network, and extracting features from the outputs of the k1th, k2th...kkth...knth intermediate layers of the teacher network; then inputting the training images into a student network, and extracting features from the outputs of the s1th, s2th...skth...snth intermediate layers of the student network; wherein s1th, s2th...kkth...snth and k1th, k2th...skth...knth can be the same or different; and targeting the features from the outputs of the k1th, k2th...kkth...knth intermediate layers of the teacher network and... The features output by the s1, s2, ..., sk, ..., sn-th network nodes in the student network are as follows: The features output by the kk-th intermediate layer of the teacher network and the features output by the sk-th intermediate layer of the student network are mapped to the latent space respectively to obtain the first and second features in the latent space. kk and sk can be the same or different. The first and second features are combined in the latent space to obtain the combined features. The combined features are mapped back to the space where the features output by the intermediate layers of the teacher network are located to obtain the mixed features. The student network is trained using the mixed features.

[0007] As can be seen, the proposed solution samples the features output by multiple intermediate layers in both the teacher and student networks. Then, it maps these features to latent spaces. In the latent space, it combines one feature from each intermediate layer of the teacher and student networks, and maps the combined feature back to the space containing the original intermediate layer features, resulting in multiple mixed features. These mixed features are then used to train the student network. Each mixed feature carries information or meanings represented or implied by both the intermediate layer features in the teacher and student networks. Training the student network based on these mixed features enables it to learn the knowledge and performance of the teacher network more effectively and quickly. The knowledge distillation method provided in this application can be applied to various super-resolution scenarios, and the student network obtained using this method achieves good results when applied to super-resolution scenarios.

[0008] Based on the first aspect, in possible implementations, the features output by the kkth intermediate layer of the teacher network and the features output by the skth intermediate layer of the student network are different in dimension, while the first and second features have the same dimension in the latent space.

[0009] The feature dimensions output by the teacher network differ from those output by the student network, and this is mapped to the latent space to achieve dimensional alignment. In one example, in the latent space, the first and second features are dimensionally aligned with the features output by the kk-th intermediate layer of the teacher network. In another example, the first and second features are dimensionally aligned with the features output by the sk-th intermediate layer of the student network. Optionally, only the first and second features need to be dimensionally aligned; the aligned dimensions can differ from both the feature dimensions output by the kk-th intermediate layer of the teacher network and the feature dimensions output by the sk-th intermediate layer of the student network.

[0010] Based on the first aspect, in a possible implementation, the features output by the kkth intermediate layer of the teacher network and the features output by the skth intermediate layer of the student network are mapped to the latent space respectively to obtain the first feature and the second feature in the latent space, including: encoding the features output by the kkth intermediate layer of the teacher network through a first encoder to obtain the first feature in the latent space; and encoding the features output by the skth intermediate layer of the student network through a second encoder to obtain the second feature in the latent space.

[0011] Because student networks differ from teacher networks in terms of network performance and the amount of data in their parameters, the features extracted by the intermediate layers of student networks may have different distributions when extracting features from training images. For example, some student networks may extract facial features better than others, such as nose features. Therefore, teacher and student networks use different encoders to map the features output by the intermediate layers into the latent space. If the same encoder is used, the performance of the trained student network will be poor. Therefore, in this application, different encoders are used for encoding the teacher and student networks, which helps to improve the performance of the student network.

[0012] Based on the first aspect, in possible implementations, the first feature and the second feature are combined in the latent space to obtain the combined feature, including: in the latent space, multiplying the first feature and the mask, multiplying the second feature and the complementary mask of the mask, and then summing the products of the two to obtain the combined feature.

[0013] It is understandable that the combined processed features carry some information and meanings represented or implied by the features output by the intermediate layer in the teacher network, as well as some information and meanings represented or implied by the features output by the intermediate layer in the student network.

[0014] Based on the first aspect, in a possible implementation, the method further includes: calculating a first loss function based on the mixed features and the features output by the kkth intermediate layer of the teacher network, the first loss function being used to update the parameters in the second encoder.

[0015] Based on the mixed feature pk and the feature output by the kkth intermediate layer of the teacher network, the error between the two can be calculated. Then, the first loss function is calculated based on this error. The first loss function is used to update the parameters in the second encoder so that the student network learns the knowledge and performance of the teacher network in feature processing.

[0016] Based on the first aspect, in a possible implementation, training the student network using hybrid features includes: randomly inputting the hybrid features into the layer following the kkth intermediate layer of the teacher network or the layer following the skth intermediate layer of the student network, continuing forward propagation to obtain an output image; calculating a second loss function based on the output image and a reference image, wherein the reference image is the image obtained by inputting the training image into the teacher network; and using the second loss function to perform backpropagation on the student network to update the parameters in the student network.

[0017] Feature fusion operations and forward propagation of fused features enable the student network to learn from the teacher network in both feature processing and structure, preserving the teacher network's performance and knowledge as much as possible. Furthermore, this application's scheme essentially sets up a selector, including teacher network options and student network options. In each round of iterative learning, the selector chooses different "paths" for the multiple fused features obtained from the fusion process, ensuring that some fused features propagate in the student network while others propagate in the teacher network. For multiple rounds of iterative learning, the selection of fused features obtained from the same intermediate layer fusion process may differ in each round. This random selectivity of the selector allows the student network to learn the teacher network's knowledge and performance better and faster.

[0018] Based on the first aspect, in a possible implementation, training the student network using hybrid features includes: not inputting the hybrid features into the teacher network or the student network, but only using the first loss function to update the parameters in the sk-th intermediate layer and the layers preceding the sk-th intermediate layer in the student network.

[0019] As can be seen, the hybrid features can also be used without being input into the teacher or student network for forward propagation. Instead, the first loss function is calculated using the hybrid features and the features output from the kk-th intermediate layer of the teacher network. This first loss function is then used for backpropagation starting from the sk-th intermediate layer of the student network, updating the parameters of the sk-th intermediate layer and the layers preceding it. This reduces the forward propagation process, saves training time for the student network, and helps improve its performance.

[0020] Based on the first aspect, in possible implementations, the method is applied to a knowledge distillation method of a single dimension, or the method is applied to a knowledge distillation method of a composite dimension; when the method is applied to a knowledge distillation method of a composite dimension, the method is applied to one of the dimensions in the composite dimension, or the method is applied to multiple dimensions in the composite dimension respectively, wherein the multiple dimensions in the composite dimension are decoupled.

[0021] It is understandable that the aforementioned alignment operation ensures that the knowledge distillation method provided by this scheme can be applied in both the depth and width dimensions.

[0022] Based on the first aspect, in a possible implementation, mapping the combined processed features back to the space containing the features output by the intermediate layer of the teacher network to obtain hybrid features includes: decoding the combined processed features through a decoder to obtain hybrid features; wherein the first encoder and decoder are pre-trained, and the training process includes: inputting the features output by the intermediate layer of the teacher network into the first encoder to obtain training encoded features; inputting the training encoded features into the decoder to obtain training decoded features; calculating a second loss function based on the training decoded features and the features output by the intermediate layer of the teacher network; and updating the parameters in the first encoder and the parameters in the decoder based on the second loss function.

[0023] Secondly, this application provides a knowledge distillation apparatus, comprising:

[0024] The first feature extraction module is used to input the training image into the trained teacher network and extract the features output by the kth intermediate layer of the teacher network.

[0025] The second feature extraction module is used to input the training image into the student network and extract the features output by the s-th intermediate layer of the student network, where s is the same as or different from k.

[0026] The feature mixing module is used to map the features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network to the latent space, respectively, to obtain the first feature and the second feature in the latent space.

[0027] The feature mixing module is further configured to combine the first feature and the second feature in the latent space to obtain the combined feature.

[0028] The feature mixing module is also used to map the combined features back to the space where the features output by the intermediate layer of the teacher network are located, to obtain mixed features;

[0029] A training module for training the student network using the hybrid features.

[0030] Based on the second aspect, in a possible implementation, the feature output by the kth intermediate layer of the teacher network is different in dimension from the feature output by the sth intermediate layer of the student network, and the first feature and the second feature have the same dimension in the latent space.

[0031] Based on the second aspect, in a possible implementation, the feature mixing module is used for:

[0032] The first feature in the latent space is obtained by encoding the feature output by the kth intermediate layer of the teacher network using the first encoder.

[0033] The second feature in the latent space is obtained by encoding the features output by the s-th intermediate layer of the student network using a second encoder.

[0034] Based on the second aspect, in a possible implementation, the feature mixing module is used for:

[0035] In the latent space, the first feature is multiplied by the mask, the second feature is multiplied by the complementary mask of the mask, and the products of the two are summed to obtain the combined feature.

[0036] Based on the second aspect, in a possible implementation, the feature mixing module is used for:

[0037] Based on the hybrid features and the features output by the kth intermediate layer of the teacher network, a first loss function is calculated, which is used to update the parameters in the second encoder.

[0038] Based on the second aspect, in a possible implementation, the training module is used for:

[0039] The mixed features are randomly input into the layer following the kth intermediate layer of the teacher network or the layer following the sth intermediate layer of the student network, and the forward propagation continues to obtain the output image;

[0040] A second loss function is calculated based on the output image and the reference image, wherein the reference image is the image obtained by inputting the training image into the teacher network;

[0041] The student network is backpropagated using the second loss function to update the parameters in the student network.

[0042] Based on the second aspect, in a possible implementation, the training module is used for:

[0043] The hybrid features are not input into the teacher network or the student network, and the parameters of the s-th intermediate layer and the layers preceding the s-th intermediate layer in the student network are updated only using the first loss function.

[0044] The functional modules of the second aspect are used to implement the methods described in the first aspect and any possible implementation of the first aspect.

[0045] Thirdly, this application provides a computing device cluster, including at least one computing device, the at least one computing device including a memory and a processor, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to implement the first aspect and any possible implementation method of the first aspect.

[0046] Fourthly, this application provides a computer storage medium including program instructions that, when executed on a cluster of computing devices, cause the cluster of computing devices to perform the first aspect and any possible implementation of the first aspect.

[0047] Fifthly, this application provides a computer program product, including program instructions that, when run on a computing device cluster, cause the computing device cluster to execute the first aspect and any possible implementation of the first aspect. Attached Figure Description

[0048] Figure 1 is a schematic diagram of the system architecture of a knowledge distillation method provided in this application;

[0049] Figure 2 is a flowchart illustrating a knowledge distillation method provided in this application;

[0050] Figure 3 is a flowchart illustrating a method for combining multiple features according to this application;

[0051] Figure 4 is a schematic diagram of a structure for combining multiple features according to this application;

[0052] Figure 5A is a schematic diagram of a performance index provided in this application;

[0053] Figure 5B is a schematic diagram of a performance index provided in this application;

[0054] Figure 6 is a schematic diagram of a data comparison result provided in this application;

[0055] Figure 7 is a schematic diagram of the structure of a system provided in this application;

[0056] Figure 8 is a schematic diagram of the structure of a computing device provided in this application;

[0057] Figure 9 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0058] Figure 10 is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0059] First, some technical terms used in this application will be explained.

[0060] Super-resolution refers to the process of generating high-resolution images from low-resolution images. A model that uses deep neural networks to improve image resolution can be called a super-resolution model. Through training on a large amount of data, the deep neural network learns the mapping relationship between low-resolution and high-resolution images, thus obtaining a super-resolution model. Inputting a low-resolution test image into the super-resolution model can produce a high-resolution image.

[0061] Latent space is a commonly used concept in machine learning and deep learning. Image features or language features are mapped to a low-dimensional space by a neural network model. The feature representations in this low-dimensional space are usually the representations of features that the neural network model has learned but cannot be directly observed. This low-dimensional space is called the latent variable space, or simply the latent space. Each point or feature in the latent space corresponds to a specific representation learned by the neural network model.

[0062] This application provides a knowledge distillation method that can be applied to the field of image processing, such as super-resolution, to improve image resolution. Specifically, it can be used for image enhancement, edge detection, image smoothing, and color correction. This application uses the application of the knowledge distillation method in the field of super-resolution as an example; this example is merely illustrative and does not constitute a limitation of this application.

[0063] First, let's introduce the system architecture of this application. See Figure 1, which is a schematic diagram of the system architecture of a knowledge distillation method provided in this application. As shown in Figure 1, the system involves a teacher network, a student network, and a feature mixing module.

[0064] The teacher network is pre-trained and can be any type of neural network, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). This application does not limit the specific type of teacher network or the training method. In this example, taking the knowledge distillation method applied to super-resolution as an example, the pre-trained teacher network can be used to improve the resolution of images. Inputting low-quality images into the teacher network can obtain high-quality images that meet the requirements.

[0065] The teacher network may include an input layer, an output layer, and multiple intermediate layers. The input layer is used to preprocess the images input to the teacher network, such as batch processing of a large number of images and normalizing the size of each image. The output layer is used to process the image output by the last intermediate layer to obtain the desired image. The specific functions of the input and output layers are not limited in this application. Multiple intermediate layers are used to extract features from the features, feature maps, or images output by the input layer. In one example, each intermediate layer in the teacher network has the same structure, and each intermediate layer extracts features from shallow to deep layers; that is, each subsequent intermediate layer further extracts features based on the features output by the previous intermediate layer. Each intermediate layer may include multiple layers, such as convolutional layers, pooling layers, deconvolutional layers, etc. When the teacher network is a different neural network, the structure of each intermediate layer is different, and each intermediate layer may include more or fewer layers. The specific structure of each intermediate layer is not limited in this application.

[0066] The student network is built upon the teacher network. Similarly, the student network includes an input layer, an output layer, and multiple intermediate layers. The input layer preprocesses the images input to the student network, such as batch processing of a large number of images and normalizing the size of each image. The output layer processes the image output by the last intermediate layer to obtain the desired image. This application does not limit the specific functions of the input and output layers. Multiple intermediate layers are used to extract features from the features, feature maps, or images output by the input layer. In one example, each intermediate layer in the student network has the same structure, and each intermediate layer extracts features from shallow to deep layers; that is, each subsequent intermediate layer further extracts features based on the features output by the previous intermediate layer. Each intermediate layer may include one or more layers, such as convolutional layers, pooling layers, deconvolutional layers, etc. When the student network is a different neural network, the structure of each intermediate layer is different, and each intermediate layer may include more or fewer layers. This application does not limit the specific structure of each intermediate layer.

[0067] It should be noted that the number of intermediate layers in the student network can be the same as the number of intermediate layers in the teacher network, for example, both containing t intermediate layers. Alternatively, the number of intermediate layers in the student network can differ from that in the teacher network, for example, the teacher network contains m intermediate layers and the student network contains n intermediate layers, where m and n are different. The number of convolutional layers, deconvolutional layers, and pooling layers in each intermediate layer of the student network can be the same as or different from that in the teacher network. The method in this application trains the student network through the teacher network, allowing the student network to retain as much of the knowledge and performance of the teacher network as possible. After training, a student network that meets the requirements is finally obtained. The obtained student network is lightweight and easy to apply on computing devices.

[0068] The feature fusion module combines features extracted from the teacher network and features extracted from the student network to obtain hybrid features. Subsequent processing is then performed on these hybrid features to enable the student network to better learn the knowledge and performance of the teacher network. Because the feature fusion module is quite complex and closely related to the context, it will not be described in detail here; please refer to the description in the method embodiments below.

[0069] The following section uses a super-resolution application scenario as an example, combined with the system architecture shown in Figure 1, to introduce the knowledge distillation method provided in this application. When introducing the knowledge distillation method provided in this application, the section explains how the student network learns the knowledge and performance of the teacher network in the system architecture shown in Figure 1, as well as the functional role of the feature mixing module. See Figure 2, which is a flowchart illustrating a knowledge distillation method provided in this application. The method includes, but is not limited to, the following descriptions.

[0070] S101. Obtain training images.

[0071] Training images are acquired and used to train the student network, enabling it to learn the knowledge and performance of the teacher network. In this embodiment, the training images can be multiple low-quality images. For example, in Figure 1, I... LR This represents a low-quality image (low-resolution image). For the sake of simplicity, only one image is shown in the diagram, but the actual training process includes multiple low-quality images.

[0072] S102. Input the training images into the trained teacher network to obtain reference images.

[0073] The training images are input into the trained teacher network to obtain reference images. These reference images are high-quality (high-resolution) images and are used to compare with images output by the student network or teacher network during subsequent student network training. A second loss function is calculated based on the comparison results, and this second loss function is used to update the parameters of each intermediate layer in the student network. For example, in Figure 1, using... The reference image is I, and the input training image is I. LR Includes multiple images, with an output reference image. It also includes multiple images. For the sake of simplicity, only one image is shown in the figure, but the actual training process includes multiple high-quality images.

[0074] S103. Extract the features of the output of the k1th, k2th...knth intermediate layers in the teacher network.

[0075] The teacher network includes multiple intermediate layers. After the training image is input into the trained teacher network, each intermediate layer extracts features in turn to obtain the reference image in S102. During this process, the features output by each intermediate layer are recorded and saved.

[0076] This application extracts features from the outputs of some or all intermediate layers in a teacher network. For example, it extracts features from the outputs of the k1-th, k2-th...kn-th intermediate layers, where k1, k2-th...kn can be any value, meaning the k1-th, k2-th...kn-th intermediate layers can be any intermediate layer. For instance, features can be extracted from the outputs of the 1st, 3rd, 4th, 7th, and 9th intermediate layers, or from the 3rd, 5th, 6th, 7th, and 8th intermediate layers. This can be understood as randomly extracting features from the outputs of several intermediate layers. This application does not limit the number of intermediate layer output features extracted.

[0077] It is understandable that each intermediate layer extracts features from the training images, which means extracting features from multiple images input to the teacher network, and the output features of each intermediate layer include features from multiple images.

[0078] The features output by each intermediate layer can be either feature vectors or feature maps; this application does not impose any restrictions.

[0079] S104. Input the training image into the student network and extract the features output by the s1, s2...sn intermediate layers in the student network.

[0080] The student network is built upon the teacher network and includes multiple intermediate layers. The number of intermediate layers in the student network can be the same as or different from the number of intermediate layers in the teacher network. The acquired training images are input into the student network, and each intermediate layer in the student network sequentially extracts features, ultimately yielding the output image. During this process, the features output by each intermediate layer are recorded and saved.

[0081] Extract features from the outputs of some or all intermediate layers in a student network. For example, extract features from the outputs of the s1, s2, ..., sn intermediate layers in a student network. Here, s1, s2, ..., sn can be any value, that is, the s1, s2, ..., sn intermediate layers can be any intermediate layer.

[0082] In one example, the intermediate layers from which features are extracted in the student network are the same as those from the teacher network. That is, s1, s2...sn can correspond to the same values ​​as k1, k2...kn in step S103. For example, if the teacher network extracts features from the 1st, 3rd, 4th, 7th, and 9th intermediate layers, then the student network also extracts features from the 1st, 3rd, 4th, 7th, and 9th intermediate layers. If the teacher network extracts features from the 3rd, 5th, 6th, 7th, and 8th intermediate layers, then the student network also extracts features from the 3rd, 5th, 6th, 7th, and 8th intermediate layers.

[0083] In another example, the intermediate layers from which features are extracted in the student network can be different from those in the teacher network. That is, the values ​​of s1, s2...sn can be different from the values ​​of k1, k2...kn in step S103. For example, the teacher network extracts features from the outputs of the 2nd, 5th, 6th, 8th, and 10th intermediate layers, while the student network extracts features from the outputs of the 1st, 3rd, 4th, 7th, and 9th intermediate layers. Or, for example, the teacher network extracts features from the outputs of the 2nd, 4th, 7th, 9th, and 11th intermediate layers, while the student network extracts features from the outputs of the 3rd, 5th, 6th, 7th, and 8th intermediate layers.

[0084] S105. Combine the features output by the kkth intermediate layer in the teacher network and the features output by the skth intermediate layer in the student network to obtain a hybrid feature pk, where kk and sk can be the same or different.

[0085] After extracting the features from the outputs of the k1th, k2th...knth intermediate layers in the teacher network and the s1th, s2th...snth intermediate layers in the student network, the features from the k1th intermediate layer in the teacher network and the s1th intermediate layer in the student network are combined to obtain a mixed feature p1. The features from the k2th intermediate layer in the teacher network and the s2th intermediate layer in the student network are combined to obtain a mixed feature p2. The features from the knth intermediate layer in the teacher network and the snth intermediate layer in the student network are combined to obtain a mixed feature pn.

[0086] For example, in the system architecture shown in Figure 1, taking the kk-th intermediate layer in the teacher network and the sk-th intermediate layer in the student network as examples, the feature output by the kk-th intermediate layer in the teacher network is: The feature output by the sk-th intermediate layer in the student network is Will and Perform combination processing to obtain hybrid features

[0087] It is worth noting that the hybrid feature pk is obtained by combining and processing the features output by the kk-th intermediate layer in the teacher network and the sk-th intermediate layer in the student network. Therefore, this step, by combining features, results in a hybrid feature pk that carries some information and meaning represented or implied by the features output by the kk-th intermediate layer in the teacher network, as well as some information and meaning represented or implied by the features output by the sk-th intermediate layer in the student network. In subsequent steps, this hybrid feature pk continues to propagate forward in either the student network or the teacher network, enabling the student network to learn the knowledge and performance of the teacher network better and faster.

[0088] This step can be performed by the feature mixing module in the system architecture shown in Figure 1.

[0089] S106. Input the mixed feature pk into the selector. The selector has multiple options and is used to select one from the multiple options.

[0090] This step can be performed by the feature mixing module in the system architecture shown in Figure 1.

[0091] The feature blending module includes a selector with multiple options, including teacher network and student network. Optionally, the selector may also include a discard option among its multiple options.

[0092] The mixed feature pk is input into the selector, which randomly selects one from multiple options. Specifically: when mixed feature p1 is input into the selector, it selects one from multiple options; when mixed feature p2 is input into the selector, it selects one from multiple options, and so on. When mixed feature pn is input into the selector, it selects one from multiple options. For each mixed feature pk, the selector randomly selects an option. Different mixed features may select the same option or different options. For example, when mixed feature p1 is input into the selector, the selector may select the student network; when mixed feature p2 is input into the selector, the selector may select the teacher network, and so on. When mixed feature pk is input into the selector, the selector may select the discard option; when mixed feature pn is input into the selector, the selector may select the student network. For example, when the mixed feature p1 is input into the selector, the selector might select the student network; when the mixed feature p2 is input into the selector, the selector might select the student network; and so on. When the mixed feature pk is input into the selector, the selector might select the discard option; and when the mixed feature pn is input into the selector, the selector might select the teacher network. Similarly, when the mixed feature p1 is input into the selector, the selector might select the teacher network; when the mixed feature p2 is input into the selector, the selector might select the discard option; and so on. When the mixed feature pk is input into the selector, the selector might select the student option; and when the mixed feature pn is input into the selector, the selector might select the student network.

[0093] For example, in the system architecture shown in Figure 1, hybrid features are used. Input selector R k In the middle, selector R k Selected student network (selector R) k (Point to the student network option).

[0094] S106a. With the student network selected, the mixed feature pk is input into the layer after the sk-th intermediate layer in the student network, and forward propagation continues to obtain the output image qk.

[0095] For any blending feature pk, where k is an integer from 1 to n, the selector selects an option. For a given blending feature pk, if the selector selects the student network, the blending feature pk is input into the student network. Since the blending feature is obtained by combining the features output from the sk-th intermediate layer in the student network and the features output from the kk-th intermediate layer in the teacher network, when inputting the blending feature pk into the student network, it is input into the layer following the sk-th intermediate layer, and then continues to propagate forward in the student network to obtain the output image qk. Specifically, if the sk-th intermediate layer in the student network is not the last intermediate layer, then the layer following the sk-th intermediate layer is the (sk+1)-th intermediate layer, and propagation continues forward from the (sk+1)-th intermediate layer until the output layer is reached to obtain the output image. If the sk-th intermediate layer in the student network is the last intermediate layer, then the layer following the sk-th intermediate layer is the output layer, and the blending feature pk is input into the output layer to obtain the output image.

[0096] For example, in the system architecture of Figure 1, for hybrid features If the selector selects the student network, then the blended features will be selected. The (sk+1)th intermediate layer (not the last intermediate layer) is input into the student network, and then the forward propagation continues, as shown by the thick dashed line in Figure 1, to obtain the output image.

[0097] S106b. With the teacher network selected, the mixed feature pk is input into the layer after the kkth intermediate layer in the teacher network, and forward propagation continues to obtain the output image qk.

[0098] For a given blended feature pk, where k is an integer from 1 to n, when the selector selects the teacher network, the blended feature pk is input into the teacher network. Since the blended feature is obtained by combining the features output from the sk-th intermediate layer in the student network and the features output from the kk-th intermediate layer in the teacher network, when inputting the blended feature pk into the teacher network, it is input into the layer following the kk-th intermediate layer, and then continues to propagate forward in the teacher network to obtain the output image qk. Specifically, if the kk-th intermediate layer in the teacher network is not the last intermediate layer, then the layer following the kk-th intermediate layer is the (kk+1)-th intermediate layer, and propagation continues forward from the (kk+1)-th intermediate layer until the output layer is reached to obtain the output image. If the kk-th intermediate layer in the teacher network is the last intermediate layer, then the layer following the kk-th intermediate layer is the output layer, and the blended feature pk is input into the output layer to obtain the output image.

[0099] For example, in the system architecture shown in Figure 1, as indicated by the thin dashed lines, the features output from the kl-th intermediate layer of the teacher network and the sl-th intermediate layer of the student network are extracted. Here, kl and sl can be the same or different. Then, the feature mixing module combines the features output from the kl-th intermediate layer of the teacher network and the sl-th intermediate layer of the student network to obtain a mixed feature. This mixed feature is then input into a selector. If the selector selects the teacher network (neither the mixed feature nor the selector is shown in Figure 1), the mixed feature is input into the (kl+1)-th intermediate layer of the teacher network (the kl-th intermediate layer is not the last intermediate layer), allowing the mixed feature to propagate forward through the teacher network to obtain the output image.

[0100] S106c: If the discard option is selected, the mixed feature pk is discarded and forward propagation is stopped.

[0101] For a given mixed feature pk, where k is an integer from 1 to n, if the selector selects the discard option, then the mixed feature pk will not be input into any network and will not be propagated forward.

[0102] However, based on the mixed feature pk and the feature output by the kk-th intermediate layer in the teacher network, the error between the two can be calculated, thereby calculating the first loss function. The first loss function can be understood as the loss function of the feature mixing module. Using the first loss function, the parameters in the feature mixing module can be updated. Specifically, the parameters involved in the combined processing of the features output by the teacher network and the features output by the student network can be updated. Optionally, the first loss function can also be used to backpropagate the first sk intermediate layers in the student network to update the parameters in the first sk intermediate layers of the student network. Therefore, the first loss function is also used to update the parameters in the first sk intermediate layers of the student network.

[0103] Through step S106, the features output by the teacher network and the features output by the student network are combined to obtain hybrid features that carry the information and meanings represented or implied by the features output by the teacher network, as well as the information and meanings represented or implied by the features output by the student network, thus achieving feature mixing. On the other hand, for multiple hybrid features, some hybrid features are forward-propagated in the teacher network and some hybrid features are forward-propagated in the student network. This is done in each round of iterative training. After multiple rounds of iterative training, the information and meanings carried by the hybrid features are also propagated in the student network or the teacher network, which accelerates the speed at which the teacher network transfers knowledge to the student network. This allows the student network to learn the knowledge and performance of the teacher network faster and better, achieving near consistency in performance between the student network and the teacher network.

[0104] S107. Calculate the second loss function based on the multiple output images qk and the reference image.

[0105] After step S106, for those selectors that chose the discard option, no output image was obtained; for those selectors that chose either the teacher network or the student network, output images were obtained. Based on the obtained multiple output images and reference images, a second loss function is calculated. This second loss function is the loss function for the student network. Specifically, assuming the obtained multiple output images include output image q1, output image q2, ..., output image qn, the errors between output image q1 and the reference image, output image q2 and the reference image, ..., output image qn and the reference image can be calculated. Then, based on these errors, the second loss function is calculated.

[0106] For example, in the system architecture shown in Figure 1, based on the output image and reference image The error can be calculated, and then based on the output image... and reference image The errors can be calculated... Finally, the errors between all output images and the reference image are calculated, and then the second loss function is calculated based on these errors.

[0107] S108. Use the second loss function to backpropagate the student network and update the parameters in the student network.

[0108] After calculating the second loss function, the student network is backpropagated using the second loss function. During the backpropagation process, the parameters of each intermediate layer in the student network are updated.

[0109] It should be noted that the above steps describe one iterative training process. After S108, steps S104 to S108 are executed repeatedly. That is, after the parameters in the student network change, the training samples are input into the student network again. Then, the features output by the s1th, s2th...snth intermediate layer are extracted and combined with the features output by the k1th, k2th...knth intermediate layer extracted from the teacher network (the teacher network is pre-trained and its parameters do not change) to obtain multiple mixed features. For each mixed feature, the selector gives a selection, and then step S106 is executed according to the selection to obtain multiple output images. Then, based on the error between the multiple output images and the reference image, the second loss function is calculated. Backpropagation is performed again using the second loss function to update the parameters in the student network... This process is repeated iteratively from S104 to S108 until a student network that meets the requirements is finally obtained.

[0110] Alternatively, multiple batches of training images can be used, with each batch containing multiple low-quality images. For each batch of training images, the iterative training described in the previous paragraph is performed. After training with multiple batches of training images, a student network that meets the requirements is finally obtained.

[0111] Optionally, the conditions for the end of student network training can be reaching a preset number of iterations, the student network tending to converge, or the second loss function meeting preset requirements, such as the value of the second loss function reaching a preset threshold, etc. This application does not limit the conditions for the end of student network training.

[0112] After multiple rounds of iterative training, a student network that meets the requirements was finally obtained. The student network retains the performance of the teacher network as much as possible, while also being lightweight and easy to deploy on computing devices.

[0113] As can be seen, in this application's scheme, features output from multiple intermediate layers in the teacher network and student network are sampled. These features are then mixed to obtain hybrid features. The hybrid features carry information and meanings represented or implied by the features output from the intermediate layers in both the teacher and student networks. These hybrid features are then propagated forward in either the teacher or student network, enabling the student network to learn the knowledge and performance of the teacher network more effectively and quickly. This application's scheme incorporates a selector. In each round of iterative learning, the selector chooses different "paths" for the multiple hybrid features obtained through the combination process. This ensures that some hybrid features propagate in the student network while others propagate in the teacher network. For multiple rounds of iterative learning, the selection of the same hybrid features obtained from the intermediate layer combination process may differ in each round. This randomness of the selector allows the student network to learn the knowledge and performance of the teacher network more effectively and quickly. In summary, the feature fusion operation, the selectioner settings, and the forward propagation of the fused features enable the student network to learn from the teacher network in both feature processing and structure, thereby preserving the teacher network's performance and knowledge as much as possible. The knowledge distillation method provided in this application can be applied to various super-resolution scenarios, and the student network obtained using this application achieves good results when applied to super-resolution scenarios.

[0114] This application also provides a method for combining multiple features. This method can be applied to step S105 in the above method embodiment, that is, combining the feature output by the kkth intermediate layer in the teacher network and the feature output by the skth intermediate layer in the student network to obtain a mixed feature pk. Referring to Figure 3, Figure 3 is a flowchart illustrating a method for combining multiple features provided in this application. The method includes, but is not limited to, the following description.

[0115] The method in this embodiment can be executed by the feature mixing module in the system architecture shown in Figure 1. For ease of understanding, the description will be based on the feature output by the kkth intermediate layer in the teacher network and the feature output by the skth intermediate layer in the student network. The features output by other intermediate layers in the teacher network and other intermediate layers in the student network in the above method embodiment can also be combined according to the method provided in this embodiment.

[0116] The method of this embodiment will now be described with reference to the structural schematic diagram shown in Figure 4.

[0117] S1051. Map the features output by the kkth intermediate layer of the teacher network to the latent space to obtain the first feature in the latent space.

[0118] The features output by the kkth intermediate layer of the teacher network are mapped to the latent space to obtain the first feature in the latent space. Specifically, the features output by the kkth intermediate layer of the teacher network are encoded by the first encoder to obtain the first feature in the latent space.

[0119] For example, as shown in Figure 4, the features output from the kkth intermediate layer in the teacher network in Figure 1... The first feature in the latent space is obtained by encoding using the first encoder.

[0120] S1052. Map the features output by the sk-th intermediate layer of the student network to the latent space to obtain the second feature in the latent space.

[0121] The feature output of the sk-th intermediate layer of the student network is mapped to the latent space to obtain the second feature in the latent space. Specifically, after obtaining the feature output of the sk-th intermediate layer from the student network, this feature is input into the second encoder, which encodes the feature to obtain the second feature in the latent space. For example, as shown in Figure 4, the feature output from the sk-th intermediate layer of the student network in Figure 1... The second feature in the latent space is obtained by encoding using a second encoder.

[0122] The feature dimensions output by the teacher network differ from those output by the student network. Mapping these dimensions to the latent space achieves dimensional alignment. In one example, by mapping the features output by the kk-th intermediate layer of the teacher network and the sk-th intermediate layer of the student network to the latent space respectively, the first and second features are dimensionally aligned with the features output by the kk-th intermediate layer of the teacher network. In another example, by mapping the features output by the kk-th intermediate layer of the teacher network and the sk-th intermediate layer of the student network to the latent space respectively, the first and second features are dimensionally aligned with the features output by the sk-th intermediate layer of the student network. Optionally, only the first and second features need to be dimensionally aligned; the aligned dimensions can differ from both the feature dimensions output by the kk-th intermediate layer of the teacher network and the feature dimensions output by the sk-th intermediate layer of the student network.

[0123] By mapping to the latent space, the features output by the intermediate layer of the teacher network are aligned with the features output by the intermediate layer of the student network in terms of dimension. The alignment operation also ensures that the knowledge distillation method provided in this scheme can be applied in both depth and width dimensions.

[0124] Because student networks differ from teacher networks in terms of network performance and the amount of data in their parameters, the features extracted by the intermediate layers of student networks may have different distributions. For example, some student networks may extract facial features better than others, such as nose features. Therefore, teacher and student networks use different encoders to map the features output by the intermediate layers into the latent space. If the same encoder is used, the performance of the trained student network will be poor.

[0125] S1053. The first feature and the second feature are combined and spliced ​​together using a mask to obtain the combined feature.

[0126] The first and second features are combined using a mask to obtain the combined feature. The mask can be randomly generated (also called a random mask) and is a matrix of the same size as the first and second features. Specifically, in the latent space, the first feature is multiplied by the mask, the second feature is multiplied by the complementary mask of the mask, and the products are summed to obtain the combined feature. The mask causes some elements of the first feature to become 0, and the complementary mask causes another part of the second feature to become 0. For example, if K represents the mask, then the combined feature = K * second feature + (1-K) * first feature. The combined feature includes some elements from the first feature and some elements from the second feature. K and 1-K can be called complementary matrices. A complementary matrix means that when an element at a certain position in K is 1, the element at the same position in 1-K is 0, and vice versa.

[0127] For example, in Figure 4, the first feature in the latent space Second feature By mixing and stitching together the masks, we can obtain the combined features.

[0128] S1054. The combined processed features are decoded by a decoder to obtain the mixed feature pk.

[0129] The combined features are then decoded using a decoder to obtain the hybrid feature pk. For example, in Figure 4, the first feature... Second feature The combined features obtained after splicing are decoded by a decoder to obtain the hybrid features.

[0130] In this embodiment, the first encoder and decoder are pre-trained. The training process of the first encoder and decoder is described below. Initially, the first encoder and decoder are established. In step S103 of the above method embodiment, the features output by the k1th, k2th...knth intermediate layers of the teacher network are obtained. These features can be used as training samples and input into the first encoder. The first encoder encodes the features in the training samples to obtain the features in the latent space. Then, the decoder decodes the features in the latent space to obtain each decoded feature. By calculating the error between each decoded feature and the features in the training samples, a third loss function is further calculated. The parameters in the first encoder and decoder are updated using the third loss function. Again, the features in the training samples are input into the first encoder for encoding, and then the encoded features are decoded. The error between the obtained decoded features and the features in the training samples, as well as the third loss function, are calculated. The parameters in the first encoder and decoder are updated again using the third loss function… After multiple rounds of iterative training, the trained first encoder and decoder are finally obtained.

[0131] During the training of the first encoder and decoder, features output from other intermediate layers in the teacher network can also be used as training samples; this application does not impose any restrictions. Here, the third loss function is the loss function used during the training of the first encoder and decoder.

[0132] After obtaining the first encoder through training, a second encoder is constructed based on it. In subsequent knowledge distillation methods, specifically in the feature mixing stage (feature mixing module), the second encoder can be trained to update its parameters. For example, based on the error between the mixed feature pk and the feature output by the kk-th intermediate layer in the teacher network, a first loss function can be calculated. This first loss function is used to update the parameters in the first sk intermediate layers of the student network (as mentioned in the above method embodiments). The first loss function is also used to update the parameters in the second encoder. For example, in Figures 1 and 4, based on the mixed features... Features output by the k-th intermediate layer in the teacher network The error between them can also be used to calculate the first loss function. The first loss function is used not only to update the parameters in the first sk intermediate layers of the student network, but also to update the parameters in the second encoder.

[0133] As can be seen, in this embodiment, the features output by the teacher network and the features output by the student network are combined and processed to obtain hybrid features. The hybrid features carry the information and meaning represented or implied by the features output by the teacher network, as well as the information and meaning represented or implied by the features output by the student network. This achieves feature mixing / combination and lays the foundation for students to learn the knowledge and performance of the teacher network.

[0134] The knowledge distillation method provided in this application can be applied to compression in a single dimension, that is, to compress the teacher network in a single dimension to obtain a student network that meets the requirements. The single dimension can be, for example, the depth of the neural network or the width of the neural network.

[0135] Here, depth refers to the number of layers in the neural network structure. For example, the teacher network consists of a network structure of 6 layers: convolutional layer, pooling layer, convolutional layer, pooling layer, deconvolutional layer, and pooling layer. Using the knowledge distillation method of this application, the teacher network is compressed / distilled in terms of depth to obtain a student network that meets the requirements. The obtained student network consists of a network structure of 4 layers: convolutional layer, pooling layer, deconvolutional layer, and pooling layer. Compared with the teacher network, the number of layers in the student network structure has been compressed / distilled.

[0136] Width refers to the feature dimension of a neural network, such as the feature dimension when the neural network samples features, the feature dimension when outputting features, or the feature dimension when processing features. For example, when the teacher network processes an image, the feature dimension of the feature is 20 when extracting, processing, or outputting features. Using the knowledge distillation method of this application, the teacher network is distilled / compressed in terms of width to obtain a student network that meets the requirements. When the student network is used to process images, the feature dimension of the feature is 15 when extracting, processing, or outputting features. Compared to the teacher network, the student network compresses / distills the feature dimension.

[0137] The knowledge distillation method described in this application can also be applied to the compression of composite dimensions, including the depth and width of the neural network. When the knowledge distillation method described in this application is applied to the compression of composite dimensions, the multiple dimensions in the composite dimension can be decoupled or orthogonal. Knowledge distillation can be performed on the teacher network in stages. For example, firstly, the teacher network can be compressed in depth using the method of this application to obtain an intermediate network. Then, the intermediate network can be compressed in width using the knowledge distillation method of this application or existing methods to finally obtain a student network that meets the requirements. Among them, the student network has a more lightweight characteristic compared to the intermediate network. When the lightweight requirement of the student network is high, staged compression of composite dimensions can be used. The knowledge distillation method described in this application can be applied to one dimension under composite dimension compression, or it can be applied to multiple dimensions under composite dimension compression.

[0138] Applying the knowledge distillation method described in this application to the compression of composite dimensions that are decoupled or orthogonal in dimension, by performing compression / distillation in stages on the composite dimension, such as in stages on depth and width, the resulting student network has better performance than a lightweight student network obtained by directly training the teacher network in one go.

[0139] The following data illustrates the performance of the knowledge distillation method provided in this application.

[0140] Referring to Figures 5A and 5B, Figure 5A illustrates the performance metrics of student networks obtained by compressing the same teacher network using different knowledge distillation methods in terms of depth. Figure 5B illustrates the performance metrics of student networks obtained by compressing the same teacher network in multiple dimensions (depth and width). The dimension-decoupled knowledge distillation (DDKD) method uses the knowledge distillation method of this application in terms of depth, but employs existing knowledge distillation methods in terms of width. The other methods employ existing knowledge distillation methods in both depth and width. In Figures 5A and 5B, the horizontal axis represents the peak signal-to-noise ratio (PSNR). A higher PSNR value indicates better student network performance and a superior distillation method. The vertical axis represents different knowledge distillation methods.

[0141] As can be seen from Figure 5A, the performance of the student network obtained by the method of this application is better than that of the student network obtained by other methods. As can be seen from Figure 5B, under the case of composite compression, the performance index of the student network obtained by DDKD using the knowledge distillation method of this application is the best.

[0142] Enhanced Deep Residual Networks (EDSR) and Deep Residual Channel Attention Networks (RCAN) were used as teacher networks. The knowledge distillation method provided in this application was used to compress the teacher network in depth, first obtaining the teaching assistant (TA) network (intermediate network). Then, the contrastive self-distillation (CSD) method based on contrastive learning was used to compress the TA network in width, obtaining the student network. See Figure 6 for a schematic diagram. In Figure 6, Teacher represents the teacher network, TA represents the teaching assistant network, and Student represents the student network. The data for each network in terms of channel count, residual block count, residual group count, number of parameters, floating-point operations (FLOPs), and inference speed (frakespersecond, FPS) are shown.

[0143] As can be seen from the data in Figure 6, the knowledge distillation method provided in this application can be applied to the compression of complex dimensions. Moreover, regardless of whether EDSR or RCAN is used as the teacher network, the method of this application is used to compress the teacher network in depth to obtain the teaching assistant network, and then the teaching assistant network is compressed in width to obtain the student network. Compared with the teacher network, the student network achieves better performance and lightweighting.

[0144] The knowledge distillation method provided in this application has been described above in conjunction with Figures 1 to 6. The apparatus embodiments corresponding to the method embodiments are described below.

[0145] Referring to Figure 7, which is a schematic diagram of a system provided in this application, the system includes a terminal device 600 and a knowledge distillation device 700. The terminal device 600 can be an image acquisition device, such as a camera or image sensor, used to acquire training images; the terminal device 600 can also be a database storing the training images; or it can be other devices. The knowledge distillation device 700 acquires the training images from the terminal device 600 and performs knowledge distillation on the teacher network.

[0146] The following section focuses on the knowledge distillation apparatus 700, which includes:

[0147] The first feature extraction module 710 is used to input the training image into the trained teacher network and extract the features output by the kth intermediate layer of the teacher network.

[0148] The second feature extraction module 720 is used to input the training image into the student network and extract the features output by the s-th intermediate layer of the student network, where s is the same as or different from k.

[0149] The feature mixing module 730 is used to map the features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network to the latent space, respectively, to obtain the first feature and the second feature in the latent space.

[0150] The feature mixing module 730 is also used to combine the first feature and the second feature in the latent space to obtain the combined feature.

[0151] The feature fusion module 730 is also used to map the combined features back to the space of the features output by the intermediate layer of the teacher network to obtain the fused features;

[0152] Training module 740 is used to train student networks using mixed features.

[0153] In one possible implementation, the features output by the kth intermediate layer of the teacher network are different in dimension from the features output by the sth intermediate layer of the student network, while the first and second features have the same dimension in the latent space.

[0154] In a possible implementation, the feature mixing module 730 is used to: encode the features output by the kth intermediate layer of the teacher network through a first encoder to obtain a first feature in the latent space; and encode the features output by the sth intermediate layer of the student network through a second encoder to obtain a second feature in the latent space.

[0155] In a possible implementation, the feature mixing module 730 is used to: multiply the first feature and the mask in the latent space, multiply the second feature and the complementary mask of the mask, and then sum the products of the two to obtain the combined feature.

[0156] In a possible implementation, the feature mixing module 730 is used to: calculate a first loss function based on the mixed features and the features output by the kth intermediate layer of the teacher network, the first loss function being used to update the parameters in the second encoder.

[0157] In a possible implementation, the training module 740 is used to: randomly input the mixed features into the layer following the kth intermediate layer of the teacher network or the layer following the sth intermediate layer of the student network, continue forward propagation, and obtain the output image; calculate the second loss function based on the output image and the reference image, wherein the reference image is the image obtained by inputting the training image into the teacher network; and use the second loss function to perform backpropagation on the student network to update the parameters in the student network.

[0158] In a possible implementation, the training module 740 is used to: not input the mixed features into the teacher network or the student network, but only use the first loss function to update the parameters of the s-th intermediate layer in the student network and the layers preceding the s-th intermediate layer.

[0159] The first feature extraction module 710, the second feature extraction module 720, the feature mixing module 730, and the training module 740 can all be implemented in software or in hardware. For example, the implementation of the feature mixing module 730 will be described below. Similarly, the implementation of the first feature extraction module 710, the second feature extraction module 720, and the training module 740 can refer to the implementation of the feature mixing module 730.

[0160] As an example of a software functional unit, the feature hybrid module 730 may include code running on computing devices. These computing devices can be cloud service computing devices, such as servers, virtual machines, containers, etc., and may consist of one or more devices. For example, the feature hybrid module 730 may include code running on multiple computing devices. It should be noted that the multiple computing devices used to run the code can be distributed within the same region or in different regions. Furthermore, the multiple computing devices used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple availability zones (AZs).

[0161] Similarly, multiple computing devices used to run this code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to achieve interconnection between VPCs.

[0162] As an example of a hardware functional unit, the feature hybrid module 730 may include at least one computing device. Alternatively, the feature hybrid module 730 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0163] The feature hybrid module 730 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the feature hybrid module 730 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the feature hybrid module 730 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0164] It should be noted that, in other embodiments, the feature mixing module 730 can be used to execute any step in a knowledge distillation method. The first feature extraction module 710, the second feature extraction module 720, and the training module 740 can all be used to execute any step in a knowledge distillation method. The steps implemented by the first feature extraction module 710, the second feature extraction module 720, the feature mixing module 730, and the training module 740 can be specified as needed. Different steps in a knowledge distillation method are implemented by the first feature extraction module 710, the second feature extraction module 720, the feature mixing module 730, and the training module 740 respectively, so as to realize all the functions of the knowledge distillation device 700.

[0165] Referring to Figure 8, which is a schematic diagram of a computing device provided in this application, the computing device 800 may be, for example, a server, a virtual machine, a container, etc. The computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other via the bus 802. It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0166] Bus 802 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 8, but this does not imply that there is only one bus or one type of bus. Bus 802 can include pathways for transmitting information between various components of computing device 800 (e.g., memory 806, processor 804, communication interface 808).

[0167] Processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0168] The memory 806 may include volatile memory, such as random access memory (RAM). The processor 804 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0169] The memory 806 stores executable program code, and the processor 804 executes the executable program code to implement the functions of the aforementioned first feature extraction module 710, second feature extraction module 720, feature mixing module 730, and training module 740, thereby realizing a knowledge distillation method. That is, the memory 806 stores instructions for executing a knowledge distillation method.

[0170] The communication interface 808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0171] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, virtual machine, or container, such as a central server, edge server, or sidecar container.

[0172] As shown in Figure 9, which is a schematic diagram of a computing device cluster provided in this application, the computing device cluster includes at least one computing device 800. The memory 806 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing a knowledge distillation method.

[0173] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing a knowledge distillation method. In other words, a combination of one or more computing devices 800 can be used to jointly execute instructions for a knowledge distillation method.

[0174] When at least one computing device in the computing device cluster is configured to generate the knowledge distillation device 700, the memory 806 in different computing devices 800 in the computing device cluster can store different instructions, which are used to execute part of the functions of the knowledge distillation device 700 respectively. That is, the instructions stored in the memory 806 of different computing devices 800 can implement the functions of one or more modules of the first feature extraction module 710, the second feature extraction module 720, the feature mixing module 730, and the training module 740.

[0175] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 shows a schematic diagram of another computing device cluster structure. As shown in Figure 10, two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 806 in computing device 800A stores instructions for the functions of the training module 740, and the memory 806 in computing device 800B stores instructions for executing the functions of the first feature extraction module 710, the second feature extraction module 720, and the feature mixing module 730. Computing device 800A is used to acquire training images and input them into the teacher network and student network respectively for forward propagation, recording the features output by each intermediate layer in the teacher network and student network. Computing device 800B is used to extract some features output by intermediate layers in the teacher network and student network, and perform feature combination / mixing to obtain mixed features. Computing device 800A is used to input the mixed features into the student network or teacher network for forward propagation, calculate the second loss function, and then use the second loss function to perform backpropagation on the student network to update the parameters in the student network.

[0176] It should be understood that the function of computing device 800A shown in Figure 10 can also be performed by multiple computing devices 800, or a cluster of computing devices may include multiple computing devices with the same function as computing device 800A. Similarly, the function of computing device 800B can also be performed by multiple computing devices 800, or a cluster of computing devices may include multiple computing devices with the same function as computing device 800B.

[0177] This application also provides another computing device cluster. The connection relationships between the computing devices in this cluster can be similarly referenced to the connection methods of the computing device clusters described in Figures 9 and 10. The difference is that the memory 806 of one or more computing devices 800 in this cluster may store different instructions for executing a knowledge distillation method. In some possible implementations, the memory 806 of one or more computing devices 800 in this cluster may also each store partial instructions for executing a knowledge distillation method. In other words, a combination of one or more computing devices 800 can jointly execute instructions for executing a knowledge distillation method.

[0178] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a knowledge distillation method.

[0179] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device or cluster of computing devices to perform a knowledge distillation method.

[0180] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge distillation method, characterized in that, include: The training image is input into the trained teacher network, and the features output by the kth intermediate layer of the teacher network are extracted. The training image is input into the student network, and the features output by the s-th intermediate layer of the student network are extracted, where s is the same as or different from k; The features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network are mapped to the latent space respectively to obtain the first feature and the second feature in the latent space respectively. The first feature and the second feature are combined in the latent space to obtain the combined feature. The combined features are mapped back to the space of the features output by the intermediate layer of the teacher network to obtain the hybrid features; The student network is trained using the hybrid features.

2. The method according to claim 1, characterized in that, The feature output by the kth intermediate layer of the teacher network is different in dimension from the feature output by the sth intermediate layer of the student network. In the latent space, the first feature and the second feature have the same dimension.

3. The method according to claim 1 or 2, characterized in that, The features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network are mapped to the latent space, respectively, to obtain the first feature and the second feature in the latent space, including: The first feature in the latent space is obtained by encoding the feature output by the kth intermediate layer of the teacher network using the first encoder. The second feature in the latent space is obtained by encoding the features output by the s-th intermediate layer of the student network using a second encoder.

4. The method according to any one of claims 1 to 3, characterized in that, The step of combining the first feature and the second feature in the latent space to obtain the combined feature includes: In the latent space, the first feature is multiplied by the mask, the second feature is multiplied by the complementary mask of the mask, and the products of the two are summed to obtain the combined feature.

5. The method according to claim 3 or 4, characterized in that, The method further includes: Based on the hybrid features and the features output by the kth intermediate layer of the teacher network, a first loss function is calculated, which is used to update the parameters in the second encoder.

6. The method according to any one of claims 1 to 5, characterized in that, Training the student network using the hybrid features includes: The mixed features are randomly input into the layer following the kth intermediate layer of the teacher network or the layer following the sth intermediate layer of the student network, and the forward propagation continues to obtain the output image; A second loss function is calculated based on the output image and the reference image, wherein the reference image is the image obtained by inputting the training image into the teacher network; The student network is backpropagated using the second loss function to update the parameters in the student network.

7. The method according to claim 5, characterized in that, Training the student network using the hybrid features includes: The hybrid features are not input into the teacher network or the student network, and the parameters of the s-th intermediate layer and the layers preceding the s-th intermediate layer in the student network are updated only using the first loss function.

8. The method according to any one of claims 1 to 7, characterized in that, The method is applied to a single-dimensional knowledge distillation method, or the method is applied to a composite-dimensional knowledge distillation method; when the method is applied to the composite-dimensional knowledge distillation method, the method is applied to one of the dimensions in the composite dimension, or the method is applied to multiple dimensions in the composite dimension respectively, wherein the multiple dimensions in the composite dimension are decoupled.

9. A knowledge distillation apparatus, characterized in that, include: The first feature extraction module is used to input the training image into the trained teacher network and extract the features output by the kth intermediate layer of the teacher network. The second feature extraction module is used to input the training image into the student network and extract the features output by the s-th intermediate layer of the student network, where s is the same as or different from k. The feature mixing module is used to map the features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network to the latent space, respectively, to obtain the first feature and the second feature in the latent space. The feature mixing module is further configured to combine the first feature and the second feature in the latent space to obtain the combined feature. The feature mixing module is also used to map the combined features back to the space where the features output by the intermediate layer of the teacher network are located, to obtain mixed features; A training module for training the student network using the hybrid features.

10. The apparatus according to claim 9, characterized in that, The feature output by the kth intermediate layer of the teacher network is different in dimension from the feature output by the sth intermediate layer of the student network. In the latent space, the first feature and the second feature have the same dimension.

11. The apparatus according to claim 9 or 10, characterized in that, The feature mixing module is used for: The first feature in the latent space is obtained by encoding the feature output by the kth intermediate layer of the teacher network using the first encoder. The second feature in the latent space is obtained by encoding the features output by the s-th intermediate layer of the student network using a second encoder.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The feature mixing module is used for: In the latent space, the first feature is multiplied by the mask, the second feature is multiplied by the complementary mask of the mask, and the products of the two are summed to obtain the combined feature.

13. The apparatus according to claim 11 or 12, characterized in that, The feature mixing module is used for: Based on the hybrid features and the features output by the kth intermediate layer of the teacher network, a first loss function is calculated, which is used to update the parameters in the second encoder.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The training module is used for: The mixed features are randomly input into the layer following the kth intermediate layer of the teacher network or the layer following the sth intermediate layer of the student network, and the forward propagation continues to obtain the output image; A second loss function is calculated based on the output image and the reference image, wherein the reference image is the image obtained by inputting the training image into the teacher network; The student network is backpropagated using the second loss function to update the parameters in the student network.

15. The apparatus according to claim 13, characterized in that, The training module is used for: The hybrid features are not input into the teacher network or the student network, and the parameters of the s-th intermediate layer and the layers preceding the s-th intermediate layer in the student network are updated only using the first loss function.

16. A computing device cluster, characterized in that, The method includes at least one computing device, the at least one computing device comprising a memory and a processor, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to implement the method as claimed in any one of claims 1 to 8.

17. A computer storage medium, characterized in that, Includes program instructions that, when run on a computing device cluster, cause the computing device cluster to perform the method as described in any one of claims 1 to 8.

Images