Knowledge distillation method and related apparatus

Abstract

Provided in the present application are a knowledge distillation method and a related apparatus. The method is applied in the field of image processing. The knowledge distillation method comprises: acquiring a training image, inputting the training image into a trained teacher network, and extracting a feature outputted by a kth intermediate layer in the teacher network; inputting the training image into a student network, and extracting a feature outputted by an sth intermediate layer in the student network, wherein s and k are the same or different; respectively mapping to a latent space the feature outputted by the kth intermediate layer and the feature outputted by the sth intermediate layer, so as to respectively obtain a first feature and a second feature; and performing combination processing on the first feature and the second feature in the latent space, so as to obtain a feature which has been subjected to the combination processing, mapping the feature which has been subjected to the combination processing back to the space in which the feature outputted by the intermediate layer in the teacher network is located, so as to obtain a mixed feature, and on the basis of the mixed feature, training the student network. The knowledge distillation method provided in the present application has a relatively good application effect in a super-resolution application scenario.

Description

A knowledge distillation method and related device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 28, 2024, with application number 202410224957.8 and application name “A knowledge distillation method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing, and in particular to a knowledge distillation method and related devices. Background Art

[0003] Knowledge distillation (KD) is a model compression technique based on deep learning. It aims to extract knowledge from a large, complex model (often called a teacher model or teacher network) to train a small, simplified model (often called a student model or student network). The core idea of ​​knowledge distillation is to transfer knowledge from the teacher model to the student model, making the student model lightweight and efficient while preserving the knowledge and performance of the teacher model as much as possible. Knowledge distillation compression methods have been widely used in fields such as natural language processing and image processing.

[0004] At present, common knowledge distillation methods such as feature-affinity based knowledge distillation for efficient image super-resolution (FAKD) and contrastive self-distillation (CSD) are not very effective in some super-resolution (SR) scenarios due to the characteristics of the knowledge distillation algorithm itself. Summary of the Invention

[0005] This application provides a knowledge distillation method and related devices, which can be applied to a variety of super-resolution scenarios and can achieve good results in different super-resolution scenarios.

[0006] In a first aspect, the present application provides a knowledge distillation method, comprising: obtaining a training image; inputting the training image into a trained teacher network, extracting features of the k1th, k2th…kkth…knth intermediate layer outputs of the teacher network; then inputting the training image into a student network, extracting features of the s1th, s2th…skth…snth intermediate layer outputs of the student network; wherein s1, s2…kkth…sn and k1, k2…skth…knth may be the same or different; for the features of the k1th, k2th…kkth…knth intermediate layer outputs in the teacher network and Features output by the s1th, s2th…skth…snth network nodes in the student network: map the features output by the kkth intermediate layer of the teacher network and the features output by the skth intermediate layer of the student network to the latent space respectively, and obtain the first feature and the second feature in the latent space respectively, kk and sk can be the same or different; combine the first feature and the second feature in the latent space to obtain the combined feature; map the combined feature back to the space where the features output by the intermediate layer of the teacher network are located to obtain the mixed feature; use the mixed feature to train the student network.

[0007] It can be seen that the present application solution samples the features of the outputs of multiple intermediate layers in the teacher network and the features of the outputs of multiple intermediate layers in the student network, and then maps the features of the outputs of each intermediate layer of the teacher network and the features of the outputs of each intermediate layer of the student network to the latent space respectively. In the latent space, the features output by one of the intermediate layers of the teacher network and the features output by one of the intermediate layers of the student network are combined and processed, and then the combined features are mapped back to the space where the features of the outputs of the intermediate layers of the teacher network are located, to obtain multiple mixed features, and the student network is trained using the multiple mixed features. Each mixed feature carries some information and meaning represented or implied by the features output by the intermediate layers in the teacher network and some information and meaning represented or implied by the features output by the intermediate layers in the student network. The student network is trained based on the mixed features, so that the student network can learn the knowledge and performance of the teacher network better and faster. The knowledge distillation method provided in the present application can be applied to a variety of super-resolution scenarios. When the student network obtained by the present application is applied to super-resolution scenarios, better results can be obtained.

[0008] Based on the first aspect, in a possible implementation, the feature output by the kkth intermediate layer of the teacher network is different from the feature output by the skth intermediate layer of the student network in dimension, and the first feature and the second feature have the same dimension in the latent space.

[0009] The feature dimension output by the teacher network is different from the feature dimension output by the student network, and the mapping to the latent space achieves alignment in dimension. In one example, in the latent space, the first feature and the second feature are aligned in dimension with the feature output by the kkth intermediate layer of the teacher network. In another example, in the latent space, the first feature and the second feature are aligned in dimension with the feature output by the skth intermediate layer of the student network. Optionally, it is sufficient to align the first feature and the second feature in dimension, and the aligned dimension can be different from the feature dimension output by the kkth intermediate layer of the teacher network, and different from the feature dimension output by the skth intermediate layer of the student network.

[0010] Based on the first aspect, in a possible implementation method, the features output by the kkth intermediate layer of the teacher network and the features output by the skth intermediate layer of the student network are respectively mapped to the latent space to obtain the first feature and the second feature in the latent space respectively, including: encoding the features output by the kkth intermediate layer of the teacher network through a first encoder to obtain the first feature in the latent space; encoding the features output by the skth intermediate layer of the student network through a second encoder to obtain the second feature in the latent space.

[0011] Since the student network differs from the teacher network in terms of network performance, the amount of data of parameters in the network, etc., when the middle layer of the student network extracts features of the training image, the extracted features may be distributed differently. For example, some facial features are extracted better, some nose features are extracted better, etc. Therefore, the teacher network and the student network use different encoders to map the features output by the middle layer to the latent space. If the same encoder is used, the performance of the trained student network is not good. Therefore, in the present application scheme, different encoders are used for encoding the teacher network and the student network, which helps to improve the performance of the student network.

[0012] Based on the first aspect, in a possible implementation method, the first feature and the second feature are combined in the latent space to obtain the combined feature, including: in the latent space, multiplying the first feature and the mask, multiplying the second feature and the complementary mask of the mask, and then summing the products of the two to obtain the combined feature.

[0013] It can be understood that the combined processed features carry some information and meanings represented or implied by the features output by the middle layer in the teacher network and some information and meanings represented or implied by the features output by the middle layer in the student network.

[0014] Based on the first aspect, in a possible implementation, the method further includes: calculating a first loss function based on the mixed features and the features output by the kkth intermediate layer of the teacher network, and the first loss function is used to update the parameters in the second encoder.

[0015] Based on the mixed feature pk and the features output by the kkth intermediate layer of the teacher network, the error between the two can be calculated, and then the first loss function is calculated based on the error. The first loss function is used to update the parameters in the second encoder so that the student network can learn the knowledge and performance of the teacher network in feature processing.

[0016] Based on the first aspect, in a possible implementation method, the student network is trained using mixed features, including: randomly inputting the mixed features into the next layer of the kk-th intermediate layer of the teacher network or the next layer of the sk-th intermediate layer of the student network, continuing forward propagation to obtain an output image; calculating a second loss function based on the output image and a reference image, wherein the reference image is an image obtained by inputting the training image into the teacher network; and using the second loss function to backpropagate the student network to update the parameters in the student network.

[0017] The feature mixing operation and the forward propagation of the mixed features enable the student network to learn from the teacher network in terms of feature processing and structure, so as to retain the performance and knowledge of the teacher network as much as possible. In addition, the present application solution is equivalent to setting up a selector, which includes teacher network options and student network options. In a round of iterative learning, the selector will select different "lines" for the multiple mixed features obtained by the combined processing, so that some of the multiple mixed features are propagated in the student network, and some are propagated in the teacher network. For multiple rounds of iterative learning, the mixed features obtained by the same intermediate layer combined processing may also be different in each round. The random selectivity of the selector enables the student network to learn the knowledge and performance of the teacher network better and faster.

[0018] Based on the first aspect, in a possible implementation, the student network is trained using hybrid features, including: not inputting the hybrid features into the teacher network or the student network, and only using the first loss function to update the parameters in the skth intermediate layer and the layer before the skth intermediate layer in the student network.

[0019] As can be seen, the mixed features can also be used without inputting them into the teacher network or the student network for forward propagation. Instead, the mixed features and the features output by the kkth intermediate layer of the teacher network are used to calculate the first loss function. This first loss function is then used to start backpropagation at the skth intermediate layer of the student network, updating the parameters of the skth intermediate layer and the layers before the skth intermediate layer in the student network. This reduces the forward propagation process, saves student network training time, and helps improve the performance of the student network.

[0020] Based on the first aspect, in a possible implementation, the method is applied to a knowledge distillation method of a single dimension, or the method is applied to a knowledge distillation method of a composite dimension; when the method is applied to a knowledge distillation method of a composite dimension, the method is applied to one of the dimensions in the composite dimension, or the method is applied to multiple dimensions in the composite dimension respectively, wherein the multiple dimensions in the composite dimension are decoupled.

[0021] It can be understood that the aforementioned alignment operation ensures that the knowledge distillation method provided by this solution can be applied in both depth and width dimensions.

[0022] Based on the first aspect, in a possible implementation method, the combined processed features are mapped back to the space where the features output by the intermediate layer of the teacher network are located to obtain mixed features, including: decoding the combined processed features through a decoder to obtain mixed features; wherein the first encoder and decoder are pre-trained, and the training process includes: inputting the features output by the intermediate layer of the teacher network into the first encoder to obtain training encoding features; inputting the training encoding features into the decoder to obtain training decoding features; calculating the second loss function based on the training decoding features and the features output by the intermediate layer of the teacher network; and updating the parameters in the first encoder and the parameters in the decoder based on the second loss function.

[0023] In a second aspect, the present application provides a knowledge distillation device, comprising:

[0024] A first feature extraction module is used to input the training image into the trained teacher network and extract the features output by the kth intermediate layer of the teacher network;

[0025] A second feature extraction module is used to input the training image into the student network and extract the features output by the s-th intermediate layer of the student network, where s is the same as or different from k;

[0026] A feature mixing module is used to map the features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network to a latent space, respectively, to obtain a first feature and a second feature in the latent space;

[0027] The feature mixing module is further configured to combine the first feature and the second feature in the latent space to obtain a combined feature;

[0028] The feature mixing module is further used to map the combined features back to the space where the features output by the middle layer of the teacher network are located to obtain mixed features;

[0029] A training module is used to train the student network using the mixed features.

[0030] Based on the second aspect, in a possible implementation, the features output by the kth intermediate layer of the teacher network are different in dimension from the features output by the sth intermediate layer of the student network, and in the latent space, the first features and the second features have the same dimension.

[0031] Based on the second aspect, in a possible implementation, the feature mixing module is used to:

[0032] Encoding the features output by the kth intermediate layer of the teacher network through a first encoder to obtain a first feature in the latent space;

[0033] The features output by the sth intermediate layer of the student network are encoded by a second encoder to obtain the second features in the latent space.

[0034] Based on the second aspect, in a possible implementation, the feature mixing module is used to:

[0035] In the latent space, the first feature and the mask are multiplied together, the second feature and the complementary mask of the mask are multiplied together, and the products of the two are summed to obtain the combined feature.

[0036] Based on the second aspect, in a possible implementation, the feature mixing module is used to:

[0037] A first loss function is calculated based on the mixed features and the features output by the kth intermediate layer of the teacher network, where the first loss function is used to update the parameters in the second encoder.

[0038] Based on the second aspect, in a possible implementation, the training module is used to:

[0039] Randomly inputting the mixed features into the next layer of the kth intermediate layer of the teacher network or the next layer of the sth intermediate layer of the student network, continuing forward propagation, and obtaining an output image;

[0040] Calculating a second loss function based on the output image and a reference image, wherein the reference image is an image obtained by inputting the training image into the teacher network;

[0041] Backpropagation is performed on the student network using the second loss function to update parameters in the student network.

[0042] Based on the second aspect, in a possible implementation, the training module is used to:

[0043] The mixed features are not input into the teacher network or the student network, and only the first loss function is used to update the parameters of the sth intermediate layer and the layer before the sth intermediate layer in the student network.

[0044] The functional modules of the second aspect are used to implement the method described in the first aspect and any possible implementation manner of the first aspect.

[0045] In a third aspect, the present application provides a computing device cluster comprising at least one computing device, wherein the at least one computing device comprises a memory and a processor, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to implement the above-mentioned first aspect and any possible implementation method of the first aspect.

[0046] In a fourth aspect, the present application provides a computer storage medium comprising program instructions, which, when executed on a computing device cluster, enables the computing device cluster to execute the above-mentioned first aspect and any possible implementation method of the first aspect.

[0047] In a fifth aspect, the present application provides a computer program product, comprising program instructions. When the program instructions are executed on a computing device cluster, the computing device cluster executes the above-mentioned first aspect and any possible implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG1 is a schematic diagram of the system architecture of a knowledge distillation method provided by this application;

[0049] FIG2 is a schematic diagram of a process of a knowledge distillation method provided by this application;

[0050] FIG3 is a flow chart of a method for combining multiple features provided by the present application;

[0051] FIG4 is a schematic diagram of a structure for combining multiple features provided by the present application;

[0052] FIG5A is a schematic diagram of a performance indicator provided by this application;

[0053] FIG5B is a schematic diagram of a performance indicator provided by this application;

[0054] FIG6 is a schematic diagram of a data comparison result provided by this application;

[0055] FIG7 is a schematic structural diagram of a system provided by the present application;

[0056] FIG8 is a schematic diagram of the structure of a computing device provided by the present application;

[0057] FIG9 is a schematic diagram of the structure of a computing device cluster provided by the present application;

[0058] FIG10 is a schematic diagram of the structure of another computing device cluster provided in this application. DETAILED DESCRIPTION

[0059] First, some technical terms involved in this application are explained.

[0060] Super-resolution refers to the process of generating a high-resolution image from a low-resolution image. Models that use deep neural networks to improve image resolution are called super-resolution models. Through extensive training with data, deep neural networks learn the mapping relationship between low-resolution and high-resolution images, generating a super-resolution model. By inputting a low-resolution test image into the super-resolution model, a high-resolution image can be generated.

[0061] Latent space is a common concept in machine learning and deep learning. Image or language features are mapped into a low-dimensional space by a neural network model. The feature representations in this low-dimensional space are typically representations of features learned by the neural network model that cannot be directly observed. This low-dimensional space is called the latent variable space, or simply the latent space. Each point or feature in the latent space corresponds to a specific representation learned by the neural network model.

[0062] This application provides a knowledge distillation method that can be applied to image processing, such as super-resolution, to improve image resolution. Specifically, it can be used for image enhancement, edge detection, image smoothing, color correction, and other aspects. This application scheme introduces the knowledge distillation method using the super-resolution field as an example. This example is merely illustrative and does not constitute a limitation of this application scheme.

[0063] First, let's introduce the system architecture of this application. See Figure 1, which is a schematic diagram of the system architecture of a knowledge distillation method provided by this application. As shown in Figure 1, the system involves a teacher network, a student network, and a feature mixing module.

[0064] The teacher network is pre-trained and can be any type of neural network, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). This application does not limit the specific type of teacher network or its training method. In this example, the knowledge distillation method is applied to super-resolution. The trained teacher network can be used to improve the resolution of the image. By inputting low-quality images into the teacher network, a high-quality image that meets the requirements can be obtained.

[0065] The teacher network may include an input layer, an output layer and multiple intermediate layers. The input layer is used to pre-process the images input to the teacher network, for example, including batch processing of a large number of images input to the teacher network and normalizing the size of each image, etc. The output layer is used to process the image output by the last intermediate layer to obtain the desired image. The specific functions of the input layer and the output layer are not limited in this application. The multiple intermediate layers are respectively used to extract features from the features or feature maps or images output by the input layer. In one example, the structure of each intermediate layer in the teacher network is the same, and each intermediate layer extracts features from shallow to deep, that is, the latter intermediate layer further extracts features based on the features output by the previous intermediate layer. Each intermediate layer may include multiple layers, such as convolutional layers, pooling layers, deconvolution layers, pooling layers, etc. When the teacher network is a different neural network, the structure of each intermediate layer is different, and each intermediate layer may also include more or fewer layers. This application does not limit the specific structure of each intermediate layer.

[0066] The student network is established based on the teacher network. Similarly, the student network includes an input layer, an output layer and multiple intermediate layers. The input layer is used to preprocess the images input to the student network, for example, including batch processing of a large number of images input to the student network and normalizing the size of each image, etc. The output layer is used to process the image output by the last intermediate layer to obtain the desired image. The specific functions of the input layer and the output layer are not limited in this application. The multiple intermediate layers are respectively used to extract features from the features or feature maps or images output by the input layer. In one example, the structure of each intermediate layer in the student network is the same, and each intermediate layer extracts features from shallow to deep, that is, the latter intermediate layer further extracts features based on the features output by the previous intermediate layer. Each intermediate layer may include one or more layers, for example, each intermediate layer may include one or more layers of convolutional layers, pooling layers, deconvolution layers, pooling layers, etc. When the student network is a different neural network, the structure of each intermediate layer is different, and each intermediate layer may also include more or fewer layers. This application does not limit the specific structure of each intermediate layer.

[0067] It should be noted that the number of intermediate layers included in the student network can be the same as the number of intermediate layers included in the teacher network. For example, both include t intermediate layers. The number of intermediate layers included in the student network can also be different from the number of intermediate layers included in the teacher network. For example, the teacher network includes m intermediate layers and the student network includes n intermediate layers, where m is different from n. The number of convolutional layers, deconvolutional layers, and pooling layers included in each intermediate layer in the student network can be the same as or different from the number of convolutional layers, deconvolutional layers, and pooling layers included in each intermediate layer in the teacher network. The method of the present application trains the student network through the teacher network so that the student network retains the knowledge and performance of the teacher network as much as possible. After training, a student network that meets the requirements is finally obtained. The obtained student network has the characteristics of being lightweight and is easy to be applied on computing devices.

[0068] The feature blending module is used to combine features extracted from the teacher network and features extracted from the student network to obtain blended features. The blended features are then subsequently processed to enable the student network to better learn the knowledge and performance of the teacher network. Because the functionality of the feature blending module is complex and closely tied to the context, it is not convenient to explain it in detail here. For details, please refer to the description in the method embodiment below.

[0069] The following takes the super-resolution application scenario as an example and introduces the knowledge distillation method provided by this application in combination with the system architecture of Figure 1. When introducing the knowledge distillation method provided by this application, it is introduced how the student network in the system architecture of Figure 1 learns the knowledge and performance of the teacher network, as well as the functional role of the feature mixing module. As shown in Figure 2, Figure 2 is a flow chart of a knowledge distillation method provided by this application, and the method includes but is not limited to the description of the following content.

[0070] S101: Obtain a training image.

[0071] Obtain training images, which are used to train the student network so that the student network can learn the knowledge and performance of the teacher network. In this embodiment, the training images can be multiple low-quality images. For example, in Figure 1, I LR Represents a low-quality image (low-resolution image). For the sake of simplicity, only one image is shown in the figure, but the actual training process includes multiple low-quality images.

[0072] S102: Input the training image into the trained teacher network to obtain a reference image.

[0073] The training image is input into the trained teacher network to obtain a reference image. The reference image is a high-quality image (high-resolution image). The reference image (high-quality image) is used to compare with the image output by the student network or the teacher network during the subsequent student network training process. The second loss function is calculated based on the comparison results. The second loss function is used to update the parameters of each intermediate layer in the student network. For example, in Figure 1, using Represents the reference image, the input training image I LR Including multiple images, output reference image It also includes multiple images. For the sake of simplicity, only one image is shown in the figure, but the actual training process includes multiple high-quality images.

[0074] S103. Extract the features of the outputs of the k1th, k2th, ... knth intermediate layers in the teacher network.

[0075] The teacher network includes multiple intermediate layers. After the training image is input into the trained teacher network, each intermediate layer extracts features in turn, and finally obtains the reference image in S102. In this process, the features output by each intermediate layer are recorded and saved.

[0076] Extract the features output by some or all of the intermediate layers in the teacher network. For example, extract the features output by the k1th, k2th…knth intermediate layers in the teacher network, where k1, k2…kn can be any values, that is, the k1th, k2th…knth intermediate layers can be any intermediate layers. For example, the features output by the 1st, 3rd, 4th, 7th, and 9th intermediate layers can be extracted, or the features output by the 3rd, 5th, 6th, 7th, and 8th intermediate layers can be extracted. This can be understood as randomly extracting features output by several intermediate layers from multiple intermediate layers. This application does not limit the number of extracted intermediate layer output features.

[0077] It can be understood that each intermediate layer extracts features from the training image, which is to extract features from multiple images input into the teacher network, and the features output by each intermediate layer include features from multiple images.

[0078] The features output by each intermediate layer can be feature vectors or feature maps, which is not limited in this application.

[0079] S104: Input the training image into the student network and extract the features output by the s1th, s2th, ..., snth intermediate layers in the student network.

[0080] The student network is built based on the teacher network and includes multiple intermediate layers. The number of intermediate layers in the student network can be the same as or different from that in the teacher network. The acquired training image is input into the student network, and each intermediate layer in the student network sequentially extracts features to ultimately obtain the output image. During this process, the features output by each intermediate layer are recorded and saved.

[0081] Extract the features output by some or all of the multiple intermediate layers in the student network. For example, extract the features output by the s1th, s2th...snth intermediate layers in the student network, where s1, s2...sn can be any values, that is, the s1th, s2th...snth intermediate layers can be any intermediate layers.

[0082] In one example, the intermediate layer from which features are extracted in the student network is the same as the intermediate layer from which features are extracted in the teacher network, that is, s1, s2…sn and the k1, k2…kn values ​​in step S103 may correspond to the same. For example: if the features extracted in the teacher network are the outputs of the 1st, 3rd, 4th, 7th, and 9th intermediate layers, then the features extracted in the student network are also the outputs of the 1st, 3rd, 4th, 7th, and 9th intermediate layers; if the features extracted in the teacher network are the outputs of the 3rd, 5th, 6th, 7th, and 8th intermediate layers, then the features extracted in the student network are also the outputs of the 3rd, 5th, 6th, 7th, and 8th intermediate layers.

[0083] In another example, the intermediate layer from which features are extracted in the student network may be different from the intermediate layer from which features are extracted in the teacher network, that is, the values ​​of s1, s2…sn and k1, k2…kn in step S103 may be different. For example, the features extracted in the teacher network are the outputs of the 2nd, 5th, 6th, 8th, and 10th intermediate layers, and the features extracted in the student network are the outputs of the 1st, 3rd, 4th, 7th, and 9th intermediate layers. For another example, the features extracted in the teacher network are the outputs of the 2nd, 4th, 7th, 9th, and 11th intermediate layers, and the features extracted in the student network are the outputs of the 3rd, 5th, 6th, 7th, and 8th intermediate layers.

[0084] S105. Combine the features output by the kkth intermediate layer in the teacher network and the features output by the skth intermediate layer in the student network to obtain a mixed feature pk, where kk and sk can be the same or different.

[0085] After extracting the features output by the k1th, k2th…knth intermediate layers in the teacher network and the features output by the s1th, s2th…snth intermediate layers in the student network, the features output by the k1th intermediate layer in the teacher network and the features output by the s1th intermediate layer in the student network are combined to obtain a mixed feature p1, the features output by the k2th intermediate layer in the teacher network and the features output by the s2th intermediate layer in the student network are combined to obtain a mixed feature p2…the features output by the knth intermediate layer in the teacher network and the features output by the snth intermediate layer in the student network are combined to obtain a mixed feature pn.

[0086] For example, in the system architecture shown in Figure 1, taking the kkth intermediate layer in the teacher network and the skth intermediate layer in the student network as an example, the feature output by the kkth intermediate layer in the teacher network is The output characteristics of the skth intermediate layer in the student network are Will and Perform combination processing to obtain mixed features

[0087] It is worth noting that the hybrid feature pk is obtained by combining and processing the features output by the kkth intermediate layer in the teacher network and the features output by the skth intermediate layer in the student network. Therefore, in this step, by combining the features, the hybrid feature pk obtained carries some information and meaning represented or implied by the features output by the kkth intermediate layer in the teacher network and some information and meaning represented or implied by the features output by the skth intermediate layer in the student network. In subsequent steps, the hybrid feature pk continues to be forward propagated in the student network or the teacher network, allowing the student network to better and faster learn the knowledge and performance of the teacher network.

[0088] This step can be performed by the feature mixing module in the system architecture of Figure 1.

[0089] S106 , inputting the mixed feature pk into a selector, wherein the selector is provided with multiple options, and the selector is used to select one from the multiple options.

[0090] This step can be performed by the feature mixing module in the system architecture of Figure 1.

[0091] The feature mixing module includes a selector, wherein the selector is provided with a plurality of options, wherein the plurality of options include a teacher network and a student network. Optionally, the plurality of options of the selector may further include a discard option.

[0092] The hybrid feature pk is input into the selector, and the selector randomly selects one from multiple options. Specifically: when the hybrid feature p1 is input into the selector, the selector selects one from multiple options; when the hybrid feature p2 is input into the selector, the selector selects one from multiple options…when the hybrid feature pn is input into the selector, the selector selects one from multiple options. For each hybrid feature pk, the selector randomly selects an option. Different hybrid features may select the same or different options. For example, when the hybrid feature p1 is input into the selector, the selector may select the student network; when the hybrid feature p2 is input into the selector, the selector may select the teacher network…When the hybrid feature pk is input into the selector, the selector may choose to discard the option; when the hybrid feature pn is input into the selector, the selector may select the student network. For example, when the mixed feature p1 is input into the selector, the selector may select the student network, when the mixed feature p2 is input into the selector, the selector may select the student network... When the mixed feature pk is input into the selector, the selector may select the discard option, when the mixed feature pn is input into the selector, the selector may select the teacher network. For another example, when the mixed feature p1 is input into the selector, the selector may select the teacher network, when the mixed feature p2 is input into the selector, the selector may select the discard option... When the mixed feature pk is input into the selector, the selector may select the student option, when the mixed feature pn is input into the selector, the selector may select the student network.

[0093] For example, in the system architecture of Figure 1, the mixed features Input selector R k In the selector R k Select the student network (selector R k Point to Student Network Options).

[0094] S106a. When the student network is selected, the mixed feature pk is input into the next layer after the sk-th intermediate layer in the student network, and forward propagation is continued to obtain the output image qk.

[0095] For any hybrid feature pk, where k is an integer between 1 and n, the selector selects one option. For a hybrid feature pk, when the selector selects the student network, the hybrid feature pk is input into the student network. Since the hybrid feature is derived by combining the features output by the skth intermediate layer in the student network with the features output by the kkth intermediate layer in the teacher network, the hybrid feature pk is input into the layer following the skth intermediate layer in the student network. The forward propagation continues in the student network to obtain the output image qk. If the skth intermediate layer in the student network is not the last intermediate layer, the layer following the skth intermediate layer is the (sk+1)th intermediate layer. Forward propagation continues from the (sk+1)th intermediate layer until the output layer is reached, obtaining the output image. If the skth intermediate layer in the student network is the last intermediate layer, the layer following the skth intermediate layer is the output layer. The hybrid feature pk is then input into the output layer to obtain the output image.

[0096] For example, in the system architecture of Figure 1, for mixed features The selector selects the student network, then the mixed features Input the (sk+1)th intermediate layer in the student network (the skth intermediate layer is not the last intermediate layer), and then continue forward propagation, as shown by the thick dotted line in Figure 1, to obtain the output image

[0097] S106b. When the teacher network is selected, the mixed feature pk is input into the next layer after the kk-th intermediate layer in the teacher network, and forward propagation is continued to obtain the output image qk.

[0098] For a certain hybrid feature pk, where k is an integer between 1 and n, when the selector selects the teacher network, the hybrid feature pk is input into the teacher network. Since the hybrid feature is obtained by combining the features output by the skth intermediate layer in the student network and the features output by the kkth intermediate layer in the teacher network, when the hybrid feature pk is input into the teacher network, it is input into the layer after the kkth intermediate layer in the teacher network. The forward propagation then continues in the teacher network to obtain the output image qk. If the kkth intermediate layer in the teacher network is not the last intermediate layer, the layer after the kkth intermediate layer is the (kk+1)th intermediate layer. The forward propagation continues from the (kk+1)th intermediate layer until it reaches the output layer to obtain the output image. If the kkth intermediate layer in the teacher network is the last intermediate layer, the layer after the kkth intermediate layer is the output layer. The hybrid feature pk is then input into the output layer to obtain the output image.

[0099] For example, in the system architecture of Figure 1, as shown by the thin dotted line in Figure 1, the features output by the klth intermediate layer in the teacher network are extracted, and the features output by the slth intermediate layer in the student network are extracted, where kl and sl can be the same or different. Then, the feature mixing module combines the features output by the klth intermediate layer in the teacher network and the features output by the slth intermediate layer in the student network to obtain mixed features. The feature mixing module then inputs the mixed features into the selector. The selector selects the teacher network (the mixed features and the selector are not shown in Figure 1), and then inputs the mixed features into the (kl+1)th intermediate layer in the teacher network (the klth intermediate layer is not the last intermediate layer), so that the mixed features are forward propagated in the teacher network to obtain the output image.

[0100] S106c: When the discard option is selected, the mixed feature pk is discarded and the forward propagation is not continued.

[0101] For a certain mixed feature pk, k takes the value of an integer from 1 to n. When the selector selects the discard option, the mixed feature pk is not input into any network and no forward propagation is performed.

[0102] However, based on the mixed feature pk and the features output by the kkth intermediate layer in the teacher network, the error between the two can be calculated, thereby calculating the first loss function. The first loss function can be understood as the loss function of the feature mixing module. Using the first loss function, the parameters in the feature mixing module can be updated. Specifically, the parameters involved in the combination of the features output by the teacher network and the features output by the student in the feature mixing module can be updated. Optionally, the first sk intermediate layers in the student network can also be backpropagated according to the first loss function to update the parameters in the first sk intermediate layers in the student network. Therefore, the first loss function is also used to update the parameters in the first sk intermediate layers in the student network.

[0103] Through step S106, the features output by the teacher network and the features output by the student network are combined and processed, and the obtained hybrid features carry the information and meaning represented or implied by the features output by the teacher network, and also carry the information and meaning represented or implied by the features output by the student network, thereby realizing feature mixing; on the other hand, for multiple hybrid features, some hybrid features are forward propagated in the teacher network, and some hybrid features are forward propagated in the student network, and this is done in each round of iterative training. After multiple rounds of iterative training, the information and meaning carried by the hybrid features are also propagated in the student network or the teacher network, thereby accelerating the speed of knowledge transfer from the teacher network to the student network, enabling the student network to learn the knowledge and performance of the teacher network faster and better, and achieving near consistency in performance between the student network and the teacher network.

[0104] S107: Calculate a second loss function based on the obtained multiple output images qk and the reference image.

[0105] After S106, if the selector selects the discard option, no output image is obtained. If the selector selects the teacher network or the student network, both output images are obtained. Based on the multiple output images obtained and the reference image, a second loss function is calculated. The second loss function is the loss function of the student network. Specifically, assuming that the multiple output images obtained include output image q1, output image q2, ..., output image qn, then the error between output image q1 and the reference image, the error between output image q2 and the reference image, ..., the error between output image qn and the reference image can be calculated. Then, based on the error between output image q1 and the reference image, the error between output image q2 and the reference image, ..., the error between output image qn and the reference image, the second loss function is calculated.

[0106] For example, in the system architecture of Figure 1, according to the output image and reference images The error can be calculated and then based on the output image and reference images The error can be calculated...Finally, the error between all output images and the reference image is calculated, and then based on these errors, the second loss function is calculated

[0107] S108. Use the second loss function to perform backpropagation on the student network to update the parameters in the student network.

[0108] After calculating the second loss function, the student network is back-propagated using the second loss function. During the back-propagation process, the parameters of each intermediate layer in the student network are updated.

[0109] It should be noted that the above steps describe the process of one iterative training. After S108, steps S104 to S108 are executed in a loop, that is, after the parameters in the student network change, the training samples are input into the student network again, and then the features output by the s1th, s2th...snth intermediate layers are extracted and combined with the features output by the k1th, k2th...knth intermediate layers extracted from the teacher network (the teacher network is trained and the parameters in the teacher network do not change) to obtain multiple mixed features. For each mixed feature, the selector gives a selection, and then step S106 is executed according to the selection to obtain multiple output images. Then, based on the errors between the multiple output images and the reference image, the second loss function is calculated, and the second loss function is used to perform back propagation again to update the parameters in the student network... S104 to S108 are executed in such an iterative loop to finally obtain a student network that meets the requirements.

[0110] Optionally, multiple batches of training images can be used, each batch of training images includes multiple low-quality images. For each batch of training images, the iterative training described in the previous paragraph is performed. After training on multiple batches of training images, a student network that meets the requirements is finally obtained.

[0111] Optionally, the condition for the end of student network training may be reaching a preset number of iterations, or the student network tends to converge, or the second loss function meets preset requirements, such as the value of the second loss function reaches a preset threshold, etc. This application does not limit the conditions for the end of student network training.

[0112] After multiple rounds of iterative training, a student network that meets the requirements is finally obtained. The student network retains the performance of the teacher network as much as possible, while also being lightweight and easy to deploy on computing devices.

[0113] It can be seen that in the present application scheme, by sampling the features output by multiple intermediate layers in the teacher network and the features output by multiple intermediate layers in the student network, the features output by the intermediate layers in the teacher network are then mixed with the features output by the intermediate layers in the student network to obtain mixed features. The mixed features carry some information and meanings represented or implied by the features output by the intermediate layers in the teacher network and some information and meanings represented or implied by the features output by the intermediate layers in the student network. The mixed features are further forward propagated in the teacher network or the student network, so that the student network can better and faster learn the knowledge and performance of the teacher network. A selector is set in the present application scheme. In a round of iterative learning, the selector selects different "lines" for the multiple mixed features obtained by the combined processing, so that some of the mixed features are propagated in the student network and some are propagated in the teacher network. For multiple rounds of iterative learning, the mixed features obtained by the same intermediate layer combined processing may be selected differently in each round. The randomness of the selector enables the student network to better and faster learn the knowledge and performance of the teacher network. In summary, the feature mixing operation, selector setting, and forward propagation of the mixed features enable the student network to learn from the teacher network in terms of both feature processing and structure, thereby preserving the performance and knowledge of the teacher network as much as possible. The knowledge distillation method provided in this application can be applied to a variety of super-resolution scenarios. When the student network obtained in this application is applied to super-resolution scenarios, good results can be achieved.

[0114] The present application also provides a method for combining multiple features, which can be applied in step S105 of the above method embodiment. That is, the feature output by the kkth intermediate layer in the teacher network and the feature output by the skth intermediate layer in the student network are combined to obtain a mixed feature pk. Referring to Figure 3, Figure 3 is a flow chart of a method for combining multiple features provided by the present application, which includes but is not limited to the description below.

[0115] The method of this embodiment can be executed by the feature mixing module in the system architecture of Figure 1. For ease of understanding, the features output by the kkth intermediate layer in the teacher network and the features output by the skth intermediate layer in the student network are used as examples for description. The features output by other intermediate layers in the teacher network and the features output by other intermediate layers in the student network in the above method embodiment can all be combined and processed according to the method provided in this embodiment.

[0116] The method of this embodiment is described below with reference to the structural diagram shown in FIG4 .

[0117] S1051. Map the features output by the kkth intermediate layer of the teacher network to the latent space to obtain the first feature in the latent space.

[0118] The feature output by the kkth intermediate layer of the teacher network is mapped to the latent space to obtain the first feature in the latent space. Specifically, the feature output by the kkth intermediate layer in the teacher network is encoded by the first encoder to obtain the first feature in the latent space.

[0119] For example, as shown in Figure 4, the feature output from the kkth intermediate layer in the teacher network in Figure 1 Encode through the first encoder to obtain the first feature in the latent space

[0120] S1052. Map the features output by the skth intermediate layer of the student network to the latent space to obtain the second feature in the latent space.

[0121] The feature output by the skth intermediate layer of the student network is mapped to the latent space to obtain the second feature in the latent space. Specifically, after obtaining the feature output by the skth intermediate layer from the student network, the feature is input into the second encoder, and the feature is encoded by the second encoder to obtain the second feature in the latent space. For example, as shown in FIG4, the feature output from the skth intermediate layer of the student network in FIG1 is Encode through the second encoder to obtain the second feature in the latent space

[0122] The feature dimension of the teacher network output is different from the feature dimension of the student network output, and mapping to the latent space achieves alignment in dimension. In one example, by mapping the feature output by the kkth intermediate layer of the teacher network and the feature output by the skth intermediate layer of the student network to the latent space respectively, in the latent space, the first feature and the second feature are aligned in dimension with the feature output by the kkth intermediate layer of the teacher network. In another example, by mapping the feature output by the kkth intermediate layer of the teacher network and the feature output by the skth intermediate layer of the student network to the latent space respectively, in the latent space, the first feature and the second feature are aligned in dimension with the feature output by the skth intermediate layer of the student network. Optionally, it is sufficient to align the first feature and the second feature in dimension, and the dimension after alignment can be different from the feature dimension output by the kkth intermediate layer of the teacher network, and different from the feature dimension output by the skth intermediate layer of the student network.

[0123] By mapping to the latent space, the features of the intermediate layer output of the teacher network are aligned with the features of the intermediate layer output of the student network in terms of dimension. The alignment operation also ensures that the knowledge distillation method provided by this solution can be applied in both depth and width dimensions.

[0124] Since the student network differs from the teacher network in terms of network performance and the amount of data for parameters in the network, when the intermediate layer of the student network extracts features of the training image, the extracted features may be distributed differently. For example, some facial features are extracted better, while others are extracted better than others. Therefore, the teacher network and the student network use different encoders to map the features output by the intermediate layer to the latent space. If the same encoder is used, the performance of the trained student network will be poor.

[0125] S1053: Mix and concatenate the first feature and the second feature through a mask to obtain a combined feature.

[0126] The first feature and the second feature are mixed and concatenated using a mask to obtain a combined feature, where the mask can be randomly generated (also called a random mask) and is a matrix of the same size as the first and second features. Specifically, in the latent space, the first feature is multiplied by the mask, and the second feature is multiplied by the complementary mask of the mask, and the products are summed to obtain the combined feature, where the mask causes some elements in the first feature to become 0, and the complementary mask causes another part of the elements in the second feature to become 0. For example, if K represents the mask, then the combined feature = K*second feature + (1-K)*first feature. The combined feature includes some elements in the first feature and some elements in the second feature, where K and 1-K can be called complementary matrices. The so-called complementary matrix means that when an element at a certain position in K is 1, the element at the same position in 1-K is 0, and when an element at a certain position in K is 0, the element at the same position in 1-K is 1.

[0127] For example, in Figure 4, the first feature in the latent space and the second feature By mixing and splicing the masks, the combined features can be obtained.

[0128] S1054: Decode the combined features through a decoder to obtain a mixed feature pk.

[0129] The combined features are decoded by the decoder to obtain the mixed feature pk. For example, in Figure 4, the first feature and the second feature The combined features obtained after mixing and splicing are decoded by the decoder to obtain mixed features

[0130] In this embodiment, the first encoder and decoder are pre-trained. The following describes the training process for the first encoder and decoder. Initially, the first encoder and decoder are established. In the above method embodiment, the features output by the k1th, k2th, …, knth intermediate layers of the teacher network are obtained in step S103. These features can be input into the first encoder as training samples. The first encoder encodes the features in the training samples to obtain features in the latent space. The decoder then decodes the features in the latent space to obtain decoded features. The error between each decoded feature and the features in the training sample is calculated, and a third loss function is further calculated. The parameters in the first encoder and decoder are then updated using the third loss function. The features in the training sample are then input into the first encoder for encoding. The encoded features are then decoded. The error between the decoded features and the features in the training sample and the third loss function are calculated. The parameters in the first encoder and decoder are again updated using the third loss function. After multiple rounds of iterative training, a trained first encoder and decoder are ultimately obtained.

[0131] In the process of training the first encoder and decoder, features output by other intermediate layers in the teacher network can also be used as training samples, which is not limited in this application. Here, the third loss function is the loss function used in the process of training the first encoder and decoder.

[0132] After the first encoder is obtained through training, a second encoder is constructed based on the first encoder. In the subsequent knowledge distillation method, specifically in the feature mixing stage (feature mixing module) of the knowledge distillation method, the second encoder can also be trained to update the parameters in the second encoder. For example, based on the error between the mixed feature pk and the feature output by the kkth intermediate layer in the teacher network, a first loss function can also be calculated. The first loss function is used to update the parameters in the first sk intermediate layers in the student network (also mentioned in the above method embodiment). The first loss function is also used to update the parameters in the second encoder. For example, in Figures 1 and 4, based on the mixed feature pk, the first loss function is used to update the parameters in the first sk intermediate layers in the student network (also mentioned in the above method embodiment). and the features output by the kkth intermediate layer in the teacher network The error between them can also be used to calculate the first loss function, which is used not only to update the parameters in the first sk intermediate layers in the student network, but also to update the parameters in the second encoder.

[0133] It can be seen that in this embodiment, the features output by the teacher network are combined and processed with the features output by the student network to obtain hybrid features. The hybrid features carry the information and meaning represented or implied by the features output by the teacher network, and also carry the information and meaning represented or implied by the features output by the student network, thereby realizing the mixing / combination of features and laying the foundation for the student network to learn the knowledge and performance of the teacher network.

[0134] The knowledge distillation method provided in this application can be applied to compression in a single dimension, that is, compressing the teacher network in a single dimension to obtain a student network that meets the requirements. For example, the single dimension can be the depth of the neural network or the width of the neural network.

[0135] Here, depth refers to the number of layers of the network structure of the neural network. For example, the teacher network is composed of a six-layer network structure of convolutional layer, pooling layer, convolutional layer, pooling layer, deconvolution layer, and pooling layer. The knowledge distillation method of this application is used to compress / distill the teacher network in depth to obtain a student network that meets the requirements. The obtained student network is composed of a four-layer network structure of convolutional layer, pooling layer, deconvolution layer, and pooling layer. Compared with the teacher network, the number of layers of the student network's network structure has been compressed / distilled.

[0136] Width refers to the feature dimension of a neural network, such as any one or more of the feature dimension when the neural network samples features, outputs features, and processes features. For example, when the teacher network processes an image, extracts features from the image, processes features, or outputs features, the feature dimension of the features is 20. The knowledge distillation method of this application is used to distill / compress the teacher network in width to obtain a student network that meets the requirements. When the student network is used to apply processing to an image, the student network extracts features from the image, processes features, or outputs features, and the feature dimension of the features is 15. Compared with the teacher network, the student network compresses / distills the features in terms of their feature dimension.

[0137] The knowledge distillation method described in the present application can also be applied to the compression of composite dimensions, which include the depth and width of the neural network. When the knowledge distillation method described in the present application is applied to the compression of composite dimensions, the multiple dimensions in the composite dimensions can be decoupled or orthogonal, and the teacher network can be subjected to knowledge distillation in stages. For example, the teacher network is first compressed in depth using the method of the present application to obtain an intermediate network, and then the knowledge distillation method of the present application or the existing method is used to compress the intermediate network in width, and finally a student network that meets the requirements is obtained. Among them, the student network is more lightweight than the intermediate network. When the lightweight requirements for the student network are high, staged composite dimension compression can be adopted. The knowledge distillation method described in the present application can be applied to one dimension under composite dimension compression, or to multiple dimensions under composite dimension compression.

[0138] The knowledge distillation method described in this application is applied to the compression of composite dimensions that are decoupled or orthogonal in dimension. By performing compression / distillation in composite dimensions in stages, for example, performing distillation / compression in stages in depth and width, the performance of the obtained student network is better than that of a lightweight student network obtained by directly training the teacher network in one go.

[0139] The following data illustrates the performance of the knowledge distillation method provided in this application.

[0140] Refer to Figures 5A and 5B. Figure 5A is a schematic diagram of the performance indicators of the student network obtained by compressing the same teacher network using different knowledge distillation methods in depth. Figure 5B is a schematic diagram of the performance indicators of the student network obtained by compressing the same teacher network in a composite dimension (depth and width). The dimensionally decoupled knowledge distillation (DDKD) method uses the knowledge distillation method of the present application in depth and the existing knowledge distillation method in width. The other methods use the existing knowledge distillation methods in both depth and width. In Figures 5A and 5B, the horizontal axis represents the peak signal-to-noise ratio (PSNR). The larger the PSNR value, the better the performance of the student network and the better the distillation method. The vertical axis represents different knowledge distillation methods.

[0141] As can be seen from Figure 5A, the performance of the student network obtained using the method of this application is better than that of the student network obtained using other methods. As can be seen from Figure 5B, under the condition of composite compression, the performance index of the student network obtained by DDKD using the knowledge distillation method of this application is the best.

[0142] Use Enhanced Deep Residual Networks (EDSR) and deep residual channel attention networks (RCAN) as teacher networks respectively, use the knowledge distillation method provided in this application to compress the teacher network in depth, first obtain the teaching assistant (TA) network (intermediate network), and then use the contrastive self-distillation (CSD) method based on contrastive learning to compress the teaching assistant network in width to obtain the student network. Referring to the schematic diagram shown in Figure 6, in Figure 6, Teacher represents the teacher network, TA represents the teaching assistant network, and Student represents the student network, and shows the data information of each network in terms of the number of channels (channel), residual block (Block), residual group (Group), number of parameters (Paraks), algorithm complexity (floating-point operations, FLOPs), and inference speed (frakes per second, FPS).

[0143] As can be seen from the data in Figure 6, the knowledge distillation method provided by this application can be applied to compression of complex dimensions. Moreover, whether EDSR or RCAN is used as the teacher network, using the method of this application to compress the teacher network in depth to obtain the teaching assistant network, and then compressing the teaching assistant network in width, the resulting student network achieves better results in terms of performance and lightness compared to the teacher network.

[0144] The above description of the knowledge distillation method provided by the present application is made in combination with Figures 1 to 6. The following introduces an apparatus embodiment corresponding to the method embodiment.

[0145] Referring to Figure 7, Figure 7 is a schematic diagram of the structure of a system provided by this application, which includes a terminal device 600 and a knowledge distillation device 700. Terminal device 600 can be an image acquisition device, such as a camera or image sensor, for acquiring training images; terminal device 600 can also be a database storing training images; or terminal device 600 can be other devices. Knowledge distillation device 700 obtains training images from terminal device 600 and performs knowledge distillation on the teacher network.

[0146] The following focuses on the introduction of the knowledge distillation device 700, which includes:

[0147] A first feature extraction module 710 is used to input the training image into the trained teacher network and extract the features output by the kth intermediate layer of the teacher network;

[0148] A second feature extraction module 720 is configured to input the training image into the student network and extract features output by the sth intermediate layer of the student network, where s is the same as or different from k;

[0149] A feature mixing module 730 is used to map the features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network to the latent space, respectively, to obtain the first feature and the second feature in the latent space;

[0150] The feature mixing module 730 is further configured to combine the first feature and the second feature in the latent space to obtain a combined feature;

[0151] The feature mixing module 730 is further used to map the combined processed features back to the space where the features output by the middle layer of the teacher network are located to obtain mixed features;

[0152] The training module 740 is used to train the student network using the mixed features.

[0153] In a possible implementation, the feature output by the kth intermediate layer of the teacher network is different in dimension from the feature output by the sth intermediate layer of the student network, and the first feature and the second feature have the same dimension in the latent space.

[0154] In a possible implementation, the feature mixing module 730 is used to: encode the features output by the kth intermediate layer of the teacher network through a first encoder to obtain a first feature in the latent space; and encode the features output by the sth intermediate layer of the student network through a second encoder to obtain a second feature in the latent space.

[0155] In a possible implementation, the feature mixing module 730 is used to: in the latent space, multiply the first feature and the mask, multiply the second feature and the complementary mask of the mask, and then sum the products of the two to obtain the combined feature.

[0156] In a possible implementation, the feature mixing module 730 is used to calculate a first loss function based on the mixed features and the features output by the kth intermediate layer of the teacher network, and the first loss function is used to update the parameters in the second encoder.

[0157] In a possible implementation, the training module 740 is used to: randomly input the mixed features into the next layer of the kth intermediate layer of the teacher network or the next layer of the sth intermediate layer of the student network, continue forward propagation, and obtain an output image; calculate a second loss function based on the output image and the reference image, wherein the reference image is the image obtained by inputting the training image into the teacher network; use the second loss function to backpropagate the student network to update the parameters in the student network.

[0158] In a possible implementation, the training module 740 is used to: not input the mixed features into the teacher network or the student network, and only use the first loss function to update the parameters in the sth intermediate layer and the layer before the sth intermediate layer in the student network.

[0159] Among them, the first feature extraction module 710, the second feature extraction module 720, the feature mixing module 730, and the training module 740 can all be implemented by software or by hardware. For example, the implementation of the feature mixing module 730 is described below using the feature mixing module 730 as an example. Similarly, the implementation of the first feature extraction module 710, the second feature extraction module 720, and the training module 740 can refer to the implementation of the feature mixing module 730.

[0160] As an example of a software functional unit, the feature mixing module 730 may include code running on a computing device. The computing device may be a computing device in a cloud service, wherein the computing device may be, for example, a server, a virtual machine, a container, etc. Further, the computing device may be one or more. For example, the feature mixing module 730 may include code running on multiple computing devices. It should be noted that the multiple computing devices used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple computing devices used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Generally, a region may include multiple availability zones AZ.

[0161] Similarly, the multiple computing devices used to run the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0162] As an example of a hardware functional unit, feature mixing module 730 may include at least one computing device. Alternatively, feature mixing module 730 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0163] The multiple computing devices included in the feature mixing module 730 can be distributed in the same region or in different regions. The multiple computing devices included in the feature mixing module 730 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the feature mixing module 730 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0164] It should be noted that, in other embodiments, the feature mixing module 730 can be used to execute any step in a knowledge distillation method, and the first feature extraction module 710, the second feature extraction module 720, and the training module 740 can all be used to execute any step in a knowledge distillation method. The steps that the first feature extraction module 710, the second feature extraction module 720, the feature mixing module 730, and the training module 740 are responsible for implementing can be specified as needed. By respectively implementing different steps in a knowledge distillation method, the full functions of the knowledge distillation device 700 can be generated.

[0165] Referring to Figure 8 , which is a schematic diagram of the structure of a computing device provided in this application, computing device 800 can be, for example, a server, a virtual machine, a container, etc. Computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. Processor 804, memory 806, and communication interface 808 communicate with each other via bus 802. It should be understood that this application does not limit the number of processors and memories in computing device 800.

[0166] Bus 802 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 802 may include a path for transmitting information between various components of computing device 800 (e.g., memory 806, processor 804, and communication interface 808).

[0167] The processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0168] The memory 806 may include volatile memory, such as random access memory (RAM). The processor 804 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0169] Memory 806 stores executable program code. Processor 804 executes the executable program code to implement the aforementioned functions of first feature extraction module 710, second feature extraction module 720, feature blending module 730, and training module 740, thereby implementing a knowledge distillation method. Specifically, memory 806 stores instructions for executing a knowledge distillation method.

[0170] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0171] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, a virtual machine, or a container, such as a central server, an edge server, or a sidecar container.

[0172] As shown in Figure 9, Figure 9 is a structural diagram of a computing device cluster provided in this application, wherein the computing device cluster includes at least one computing device 800, and the memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing a knowledge distillation method.

[0173] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing a knowledge distillation method. In other words, the combination of one or more computing devices 800 can be used to jointly execute instructions for a knowledge distillation method.

[0174] When at least one computing device in a computing device cluster is configured to generate the knowledge distillation apparatus 700, the memories 806 in different computing devices 800 in the computing device cluster may store different instructions, each for executing a portion of the functions of the knowledge distillation apparatus 700. In other words, the instructions stored in the memories 806 in different computing devices 800 may implement the functions of one or more of the first feature extraction module 710, the second feature extraction module 720, the feature mixing module 730, and the training module 740.

[0175] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG10 shows a schematic structural diagram of another computing device cluster. As shown in FIG10 , two computing devices 800A and 800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for the functions of the training module 740, and the memory 806 in the computing device 800B stores instructions for executing the functions of the first feature extraction module 710, the second feature extraction module 720, and the feature mixing module 730. The computing device 800A is used to obtain training images and input the training images into the teacher network and the student network respectively for forward propagation, record the features output by each intermediate layer in the teacher network and the student network, and the computing device 800B is used to extract the features output by some intermediate layers in the teacher network and the student network, and combine / mix the features to obtain mixed features. The computing device 800A is used to input the mixed features into the student network or the teacher network for forward propagation, and calculate the second loss function, and then use the second loss function to perform backpropagation on the student network to update the parameters in the student network.

[0176] It should be understood that the functions of computing device 800A shown in FIG10 may also be performed by multiple computing devices 800, or a computing device cluster may include multiple computing devices having the same functions as computing device 800A. Similarly, the functions of computing device 800B may also be performed by multiple computing devices 800, or a computing device cluster may include multiple computing devices having the same functions as computing device 800B.

[0177] An embodiment of the present application also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection method of the computing device cluster described in Figures 9 and 10. The difference is that the memory 806 in one or more computing devices 800 in the computing device cluster can store different instructions for executing a knowledge distillation method. In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster can also respectively store partial instructions for executing a knowledge distillation method. In other words, a combination of one or more computing devices 800 can jointly execute instructions for executing a knowledge distillation method.

[0178] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be executed on a computing device or stored in any available medium. When the computer program product is executed on at least one computing device, the at least one computing device performs a knowledge distillation method.

[0179] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device or a computing device cluster to perform a knowledge distillation method.

[0180] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge distillation method, characterized in that: include: Input the training image into the trained teacher network and extract the features output by the kth intermediate layer of the teacher network; Input the training image into the student network, and extract the features output by the sth intermediate layer of the student network, where s is the same as or different from k; Mapping the features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network to latent spaces, respectively, to obtain first features and second features in the latent spaces; Combining the first feature and the second feature in the latent space to obtain a combined feature; Mapping the combined features back to the space where the features output by the middle layer of the teacher network are located to obtain mixed features; The student network is trained using the mixed features.

2. The method according to claim 1, characterized in that The features output by the kth intermediate layer of the teacher network are different from the features output by the sth intermediate layer of the student network in dimension, and the first features and the second features have the same dimension in the latent space.

3. The method according to claim 1 or 2, characterized in that Mapping the features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network to latent spaces, respectively, to obtain first features and second features in the latent spaces, respectively, including: Encoding the features output by the kth intermediate layer of the teacher network through a first encoder to obtain a first feature in the latent space; The features output by the sth intermediate layer of the student network are encoded by a second encoder to obtain the second features in the latent space.

4. The method according to any one of claims 1 to 3, characterized in that Combining the first feature and the second feature in the latent space to obtain a combined feature includes: In the latent space, the first feature and the mask are multiplied together, the second feature and the complementary mask of the mask are multiplied together, and the products of the two are summed to obtain the combined feature.

5. The method according to claim 3 or 4, characterized in that The method further comprises: A first loss function is calculated based on the mixed features and the features output by the kth intermediate layer of the teacher network, where the first loss function is used to update the parameters in the second encoder.

6. The method according to any one of claims 1 to 5, characterized in that The method of training the student network using the mixed features comprises: Randomly inputting the mixed features into the next layer of the kth intermediate layer of the teacher network or the next layer of the sth intermediate layer of the student network, continuing forward propagation, and obtaining an output image; Calculating a second loss function based on the output image and a reference image, wherein the reference image is an image obtained by inputting the training image into the teacher network; Backpropagation is performed on the student network using the second loss function to update parameters in the student network.

7. The method according to claim 5, characterized in that The method of training the student network using the mixed features comprises: The mixed features are not input into the teacher network or the student network, and only the first loss function is used to update the parameters of the sth intermediate layer and the layer before the sth intermediate layer in the student network.

8. The method according to any one of claims 1 to 7, characterized in that The method is applied to a knowledge distillation method of a single dimension, or the method is applied to a knowledge distillation method of a composite dimension; when the method is applied to the knowledge distillation method of the composite dimension, the method is applied to one of the dimensions in the composite dimension, or the method is applied to multiple dimensions in the composite dimension respectively, wherein the multiple dimensions in the composite dimension are decoupled.

9. A knowledge distillation device, characterized in that: include: A first feature extraction module is used to input the training image into the trained teacher network and extract the features output by the kth intermediate layer of the teacher network; A second feature extraction module is used to input the training image into the student network and extract the features output by the s-th intermediate layer of the student network, where s is the same as or different from k; A feature mixing module is used to map the features output by the kth intermediate layer of the teacher network and the features output by the sth intermediate layer of the student network to a latent space, respectively, to obtain a first feature and a second feature in the latent space; The feature mixing module is further configured to combine the first feature and the second feature in the latent space to obtain a combined feature; The feature mixing module is further used to map the combined features back to the space where the features output by the middle layer of the teacher network are located to obtain mixed features; A training module is used to train the student network using the mixed features.

10. The device according to claim 9, characterized in that The features output by the kth intermediate layer of the teacher network are different from the features output by the sth intermediate layer of the student network in dimension, and the first features and the second features have the same dimension in the latent space.

11. The device according to claim 9 or 10, characterized in that The feature mixing module is used to: Encoding the features output by the kth intermediate layer of the teacher network through a first encoder to obtain a first feature in the latent space; The features output by the sth intermediate layer of the student network are encoded by a second encoder to obtain the second features in the latent space.

12. The device according to any one of claims 9 to 11, characterized in that The feature mixing module is used to: In the latent space, the first feature and the mask are multiplied together, the second feature and the complementary mask of the mask are multiplied together, and the products of the two are summed to obtain the combined feature.

13. The device according to claim 11 or 12, characterized in that The feature mixing module is used to: A first loss function is calculated based on the mixed features and the features output by the kth intermediate layer of the teacher network, where the first loss function is used to update the parameters in the second encoder.

14. The device according to any one of claims 9 to 13, characterized in that The training module is used to: Randomly inputting the mixed features into the next layer of the kth intermediate layer of the teacher network or the next layer of the sth intermediate layer of the student network, continuing forward propagation, and obtaining an output image; Calculating a second loss function based on the output image and a reference image, wherein the reference image is an image obtained by inputting the training image into the teacher network; Backpropagation is performed on the student network using the second loss function to update parameters in the student network.

15. The device according to claim 13, characterized in that The training module is used to: The mixed features are not input into the teacher network or the student network, and only the first loss function is used to update the parameters of the sth intermediate layer and the layer before the sth intermediate layer in the student network.

16. A computing device cluster, characterized in that: The method comprises at least one computing device, wherein the at least one computing device comprises a memory and a processor, and the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to implement the method according to any one of claims 1 to 8.

17. A computer storage medium, characterized in that The method comprises program instructions, which, when executed on a computing device cluster, enable the computing device cluster to execute the method according to any one of claims 1 to 8.

Images