Image super-resolution method and device based on knowledge distillation compression model

Abstract

The present invention discloses an image super-resolution method and device based on a knowledge distillation compression model, which cascades a small student network model into a teacher network with higher performance to better complete knowledge distillation, and can make the student network performance gradually approach the teacher network, thereby completing the compression of the super-resolution network. The use of the distillation strategy of the present invention not only avoids the manual design of feature transformations for alignment between different networks, but also greatly reduces the difficulty of optimizing the student network. In order to alleviate the problem of inefficient distillation caused by the representation gap between teachers and students, the present invention regards the similarity relationship between teacher layers as knowledge, so that students learn the teacher's similarity relationship in their own space, rather than directly imitating the teacher's complex features. The present invention significantly compresses the number of parameters and computational consumption of the super-resolution network model, reduces the difficulty of deploying the super-resolution network model in resource-constrained devices, and has strong practical application value.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, model compression, image super-resolution, and in particular to an image super-resolution method and device based on a knowledge distillation compression model. Background Art

[0002] Image super-resolution [1] Image super-resolution (SR) is a fundamental task in computer vision, aiming to recover a high-resolution image several times larger from a low-resolution image. Image super-resolution technology has a wide range of applications in industry, including medical image analysis, satellite image analysis, facial recognition, and surveillance. For example, when using satellites to capture terrain, power and storage limitations often result in low-resolution imaging devices. This makes terrain reconstruction difficult, and consequently, fails to meet the requirements for subsequent object recognition and analysis. In massive security surveillance scenarios, equipment costs and network bandwidth limitations result in low-resolution images, making data screening and analysis challenging. With the rise of mobile devices, people have higher demands for image quality. Within the constraints of mobile network bandwidth and device performance, obtaining pleasing high-resolution images has been a research hotspot in recent years. However, image super-resolution is a challenging and inherently imperfect problem. This is because detail information is lost when downscaling high-definition images to low-definition ones. During inverse restoration, low-definition images often have multiple high-definition mappings.

[0003] Traditional SR methods are based on manual feature extraction. Although the calculation speed is fast, the restored image is severely distorted and difficult to apply to practical scenarios. In recent years, by designing end-to-end mapping, convolutional neural networks (CNN) have achieved great success in super-resolution tasks. [2] Using only 3 layers of convolution, it achieved higher performance than traditional methods. Subsequent work focused on improving performance by using wider and deeper networks and designing efficient networks. [3] The batch normalization (BN) layer was removed and more convolutional layers were stacked to achieve better performance, which greatly influenced the subsequent mainstream SR network design work. [4] He first introduced the attention mechanism into the super-resolution task. He designed a residual group to reduce the difficulty of model training, making the network reach more than 400 layers. [5] A densely connected network was proposed. In the densely connected module, the feature maps generated by each layer are fed into the following convolutional layer to fully fuse high-level and low-level features to produce rich feature representations. However, the huge computational requirements and memory usage limit the practical industrial deployment of these networks.

[0004] On the other hand, knowledge distillation (KD), as a promising deep model compression technology, can enable a small student network to learn from a large over-parameterized teacher network, gradually approaching the performance of the teacher network, thereby enabling the small network to replace the large network for deployment. [6] First proposed in the classification task, by teaching the soft labels generated in the teacher network, the performance of the student network was greatly improved. Later, Yim et al. [7] It is proposed to regard the flow between the two layers of the teacher as knowledge and guide knowledge distillation based on the relationship between different layers. [8] It is believed that the ability difference between teachers and students is too large, which will lead to inefficiency of distillation. Recently, Jin et al. [9] Based on CKA

[10] Directly align feature maps between teacher and student for distillation.

[0005] Applying KD to the SR network can greatly reduce the dependence on computing resources while ensuring the image restoration effect, thereby making super-resolution technology widely used in practice. However, there is currently little knowledge distillation work focused on super-resolution tasks. Although some progress has been made in the distillation method of high-level visual task networks, it is very difficult to design feature alignment strategies for low-level visual tasks such as super-resolution, because the complex textures generated by the network are difficult to design strategies to align, and some form of transformation of the feature map will lead to information loss and limited effect. In order to improve the efficiency of distillation and further improve the visual restoration quality of the compressed model, it is necessary to use a new compression framework. The present invention designs a compression framework suitable for SR networks. Through this invention, the dependence of the algorithm on device resources can be greatly reduced while maintaining the image restoration effect. For example, the blurred image generated by mobile phone photography can be converted into a high-definition image through this method. Under the conditions of limited mobile phone computing resources and sensitive user waiting time, the super-resolution technology of the image can be completed quickly and with high quality, thereby meeting the user's demand for high-quality shooting.

[0006] [1].William T Freeman and Egon C Pasztor.Learning low-level vision.InICCV,1999.Long J,Shelhamer E,Darrell T.

[0007] [2].Chao Dong,Chen Change Loy,Kaiming He,and Xiaoou Tang.Image super-resolution using deep convolutional networks.IEEE transactionson patternanalysis and machine intelligence,38(2):295–307,2015.

[0008] [3].Bee Lim,Sanghyun Son,Heewon Kim,Seungjun Nah,and Kyoung MuLee.Enhanced deep residual networks for single image super-resolution.InProceedings of the IEEE conference on computer vision and pattern recognitionworkshops,pages 136–144,2017.

[0009] [4].Yulun Zhang,Kunpeng Li,Kai Li,Lichen Wang,Bineng Zhong,and YunFu.Image super-resolution using very deep residual channel attentionnetworks.In Proceedings of the European conference on computer vision(ECCV),pages 286–301,2018.

[0010] [5].Yulun Zhang,Yapeng Tian,Yu Kong,Bineng Zhong,and Yun Fu.Residualdense network for image super-resolution.In Proceedings of the IEEEconference on computer vision and pattern recognition,pages 2472–2481,2018.

[0011] [6].Geoffrey Hinton,Oriol Vinyals,and Jeff Dean.Distilling theknowledge in a neural network.arXiv preprint arXiv:1503.02531Add to Citaviproject by ArXiv ID,2015.

[0012] [7].Junho Yim,Donggyu Joo,Jihoon Bae,and Junmo Kim.A gift fromknowledge distillation:Fast optimization,network minimization and transferlearning.In Proceedings of the IEEE Conference on Computer Vision and PatternRecognition,pages 4133–4141,2017.

[0013] [8].Seyed Iman Mirzadeh,Mehrdad Farajtabar,Ang Li,Nir Levine,AkihiroMatsukawa,and Hassan Ghasemzadeh.Improved knowledge distillation via teacherassistant.In Proceedings of the AAAI Conference on Artificial Intelligence,volume 34,pages 5191–5198,2020.

[0014] [9].Qing Jin,Jian Ren,Oliver J Woodford,Jiazhuo Wang,Geng Yuan,YanzhiWang,and Sergey Tulyakov.Teachers do more than teach:Compressing image-to-image models.In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition,pages 13600–13611,2021.

[0015]

[10] . Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. InInternational Conference on Machine Learning, pages 3519–3529. PMLR, 2019. Summary of the Invention

[0016] The present invention aims to compress super-resolution network models, reducing their platform resource requirements while maintaining image restoration capabilities. To address the shortcomings of existing compression technologies in the super-resolution field, such as difficulty in feature alignment and inefficient distillation, the present invention provides an image super-resolution method and apparatus based on a knowledge distillation compression model to further enhance distillation effectiveness.

[0017] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides an image super-resolution method based on a knowledge distillation compression model, the method comprising the following steps:

[0018] (1) Obtain a large trained teacher network and a small student network to be trained, wherein the teacher network is a public, pre-trained network; the student network is a network after the teacher network is reduced in depth, which will be used for actual deployment; the teacher network and the student network are divided into two modules in units of blocks according to the network depth, respectively, to obtain teacher module 1, teacher module 2, student module 1 and student module 2; and based on the softening module that integrates the semantic features of the previous layer and the features extracted from the current layer, the teacher module 1, the softening module and the student module are cascaded in turn, and the student module 1, the softening module and the teacher module are cascaded in turn to form two optimization paths for knowledge distillation;

[0019] (2) Obtain low-definition images for the training set, where the training set is a public data set containing paired low-definition and high-definition images; input the low-definition images into two optimization paths for feature extraction; the network extracts the texture details and high-frequency features of the low-definition images as well as the structure of the image, and then generates a preliminary image result through the convolutional network. The loss term is calculated with the high-definition image to penalize the incorrect image restoration of the network; the loss term includes whether the structure is consistent, whether the color conforms to the statistical law, and whether the texture is natural. Finally, the parameters of the two paths are optimized based on the loss;

[0020] (3) The two student modules after cross-distillation are taken out and connected to each other to form the final optimization path; the low-resolution image is input to the network, and the complete teacher network is used to generate a super-resolution image for supervision, the loss term is calculated and the parameters are updated. After the training is completed, the final compression model is obtained, and the obtained low-resolution image is input into the final compression model to obtain a super-resolution image.

[0021] Furthermore, the purpose of knowledge transfer is achieved by aligning the inter-layer relationship matrices of the teacher and the student. For the output features X and Y of different layers, the specific processing flow is as follows:

[0022]

[0023]

[0024] A=CKA(X (i) ,X (j) )

[0025]

[0026] Where tr(*) represents the trace of the matrix, T represents the transpose of the matrix; A T and A S Represent the inter-layer relationship matrices of the teacher network and the student network respectively; ||*||1 represents L1 regularization.

[0027] Furthermore, a softening module is added at the cross-cascade, and k learnable parameters are used to perform linear mapping on the feature matrix to complete the dimensional matching between the teacher and the student; a softening module is designed using channel separation and residual connection. The softening module uses residuals to retain the low-frequency image contour shape to prevent image edge blurring and network gradient disappearance. The softening module is used for further extraction of high-frequency textures through 3*3 convolutional layers; it is used to smoothly transfer knowledge between the teacher network and the student network; the softening module integrates the semantic features of the previous layer with the features extracted by the current layer, serving as a transition between the teacher and the student.

[0028] Furthermore, in the cross-distillation stage, the student network is optimized by minimizing the reconstruction loss and similarity loss; in the fusion distillation stage, the student network is optimized by minimizing the reconstruction loss with the teacher.

[0029] In a second aspect, the present invention further provides an image super-resolution device based on a knowledge distillation compression model, the device comprising:

[0030] one or more processors;

[0031] a memory for storing one or more programs;

[0032] When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the image super-resolution method based on the knowledge distillation compression model.

[0033] In a third aspect, the present invention also provides a computer-readable storage medium for storing one or more computer programs, wherein the one or more computer programs include program codes. When the computer programs are run on a computer, the program codes are used to execute the image super-resolution method based on the knowledge distillation compression model.

[0034] Beneficial effects of the present invention:

[0035] (1) A new super-resolution network distillation method based on the cross-distillation paradigm is proposed. The core of this method is to directly use the parameters trained by the teacher without redesigning the feature transformation for distillation. This helps to reduce the information loss during feature transformation and improve the distillation effect.

[0036] (2) A knowledge extraction method based on inter-layer relations is proposed. The central kernel alignment method is used to ensure that the student network learns the teacher's inter-layer relation matrix in its own representation space, without directly imitating the teacher's complex representation, thereby improving the distillation effect.

[0037] (3) A soft interface module is proposed based on residual connection and channel separation operations to filter out harmful information in large models to smoothly transfer knowledge and improve the efficiency of distillation.

[0038] (4) Experiments have shown that the proposed compression method can be applied to most super-resolution networks based on deep neural networks. It not only achieves a significant reduction in the number of parameters, speeds up the calculation speed, and facilitates further industrial deployment, but also effectively maintains the performance of the super-resolution network through an efficient distillation method, and better serves downstream tasks while maintaining visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 It is the overall knowledge distillation framework of the deep neural network of the present invention.

[0041] Figure 2 Schematic diagram of the soft interface module of the present invention.

[0042] Figure 3 Examples of low-resolution images used as input for the present invention, examples of undistilled output images, and examples of images output by the present invention.

[0043] Figure 4 This is a structural diagram of the super-resolution network compression device based on knowledge distillation of the present invention. DETAILED DESCRIPTION

[0044] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0045] The primary application scenario for this invention is the compression problem of image super-resolution networks. Image super-resolution involves magnifying an input image multiple times while maintaining its subjective visual quality. SR technology can significantly improve the resolution quality of restored images without increasing hardware costs, offering high economic benefits. However, current super-resolution networks suffer from large storage space requirements, high computational overhead, and significant latency, making them difficult to deploy in practical applications such as mobile phones and edge devices, limiting their further application.

[0046] The present invention provides an image super-resolution method based on a knowledge distillation compression model, which takes a cityscape image as input and specifically includes the following steps:

[0047] 1. Problem description and variable definition

[0048] In the super-resolution of urban landscape images, the goal is to output a super-resolution image of multiple sizes based on a small-sized low-resolution image while maintaining the visual quality of the image, such as clear building outlines and rich building textures. The existing standard deep neural network-based method is: for a given input image I∈R 3* H *W , where H and W are the image height and width respectively. Input the image I into the network (such as the RCAN network YulunZhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In Proceedingsof the European conference on computer vision (ECCV), pages 286–301, 2018.), and the quadruple super-resolution network outputs the result Y0∈R 3* 4 * H *4* W.

[0049] 2. Knowledge Distillation Framework of the Present Invention

[0050] This paper designs a super-resolution network distillation framework based on cross distillation, which aims to directly use teachers for supervision and avoid explicit transformation of feature maps. Since the super-resolution network usually consists of three parts, including a head block with only one convolutional layer for shallow feature extraction; N repeated main blocks for generating high-frequency details; and an upsampling tail block for high-quality reconstruction of the final image. As shown in the attached figure, Figure 1 As shown in FIG, the cross-distillation framework of the present invention consists of two main steps. Figure 1 The cross distillation stage shown in (a) is shown in FIG. Figure 1 The fusion training stage shown in (b).

[0051] In the cross-distillation phase, the pre-trained teacher like Figure 1 (a) in the figure is divided into two parts according to the block, namely the header block and the first half of the main block connected to it. The tail block and the second half of the main block The student network has fewer main blocks than the teacher network and is divided into and Will and and The cross-cascade method obtains two new networks, forming two optimization paths. The upper optimization path is composed of and The following optimization path consists of two modules. and The network consists of two modules. The parameters of the teacher module are fixed, which supervises the student module. During training, the teacher parameters are fixed to guide the student. This allows the teacher parameters to extract specific texture and detail features during image processing. These features with less loss are then used to further guide the optimization of the student network parameters. In paired training data, this can be achieved by minimizing the following loss function:

[0052]

[0053]

[0054] in, represents a low-resolution image, Represents high-resolution images, which are paired. N represents the number of training images, and ||*||1 represents L1 regularization. Using high-definition images as constraints, the loss terms of the two paths are optimized separately, completing the first stage of distillation through supervised training. The main purpose of this stage is to decompose the student module and construct a super-resolution network using the parameters trained by the teacher to achieve distillation.

[0055] In the fusion stage, the present invention takes out the student modules trained in the previous stage from the cascade network and recombines them into the final small network. Figure 1 As shown in (b), the small network is further integrated under the supervision of the super-resolution image output by the teacher to improve the final performance of the small network. This process can be completed by minimizing the following loss under the supervision of the teacher:

[0056]

[0057] 3. Softening module (SI) of the present invention

[0058] The present invention designs a softened interface (SI) module, the purpose of which is to alleviate the ability difference between the teacher and student modules to improve distillation efficiency. Due to the difference in the number of parameters, the teacher and student modules will lead to different representation capabilities. The ability of the teacher module is much stronger than that of the student module, and this difference will lead to inefficient distillation. To alleviate this problem, the present invention designs a soft interface based on residual connection and channel separation as a transition between modules to filter out harmful information in the teacher network and improve the overall distillation efficiency. The interface is inserted between the teacher and student networks to improve the efficiency of knowledge transfer.

[0059] The overall design concept of the SI softening module is as follows: First, adjust the size of the teacher module output and use a linear embedding module E to reduce the dimension of the teacher output feature map F to obtain F0 so that it matches the input required by the student. The feature F0 is then sent to the layer-by-layer distillation sub-module to gradually refine the feature map generated by the teacher network. Finally, the refined modules of each layer are spliced ​​and fused again using a convolutional layer. The output is the feature map required by the student network. This softening interface serves as a transition between the teacher and student modules, alleviating the problem of inefficient distillation, filtering key image features such as structure, color, and texture to the student network, and filtering out some irrelevant image noise and harmful information, thereby improving the visual quality of the super-resolution image.

[0060] Specifically, as attached Figure 2 As shown, the main modules are Figure 2 In (a), the FRB submodule consists of Figure 2 As shown in (b) in the figure, the feature map generated by the teacher module is denoted as F∈R C*H*W, where C, H, w represent the tensor channel size, height and width. First, a linear embedding module E is used to reduce the dimension of F to match the input required by the student:

[0061] F0=E(F)

[0062] Where F0 represents the output, E is the 3*3 convolution with the number of input channels C and the number of output channels C / 2. Then, F0 is fed into the designed module, and multiple repeated submodules are used to gradually improve the feature extraction and gradually refine the features. Figure 2 As shown in FIG, for the input feature F0, the present invention performs two types of processing: (1) using 1x1 convolution to compress the dimension of the input channel to half of the original dimension, and directly sending the new feature F1 to the final fusion module.

[0063] F1=C0(F0)

[0064] Where C0 represents a 1x1 convolutional layer.

[0065] (2) The input F0 is sent to the submodule M0 for refinement to further generate a new distillation feature F distilled_1 .

[0066] F distilled_1 =M0(F0)

[0067] The structure of submodule M is as follows Figure 2 As shown in (b) in the figure. It consists of a 5*5 convolutional layer and a nonlinear layer ReLU. So each stage will produce two features, namely F k and F distilled_k The newly generated feature F distilled_k The next module C k and M k Further processing.

[0068] F k+1 =C k (F distilled_k ), k=1,...,n

[0069] F distilled_k+1 =M k (F distilled_k ), k=1,...,n

[0070] In the final fusion layer, all features generated by the 1×1 convolutional layer and the final distilled features are concatenated as shown below:

[0071] F all =Concat(F1,...,F k , F distilled_k ), k=1,...,n

[0072] And add them to the input feature F0 to get the refined feature map.

[0073] Finally, the feature map is reconstructed as follows.

[0074] F all =R(F all +F0)

[0075] Where R includes a 3×3 convolutional layer.

[0076] 4. Structural similarity loss of the present invention

[0077] This paper designs a behavioral similarity loss function, the purpose of which is to add finer-grained supervision to enhance the distillation effect. Based on the central kernel alignment method, this paper defines the relationship between network layers as knowledge. The purpose of distillation is achieved by aligning the similarity matrix while avoiding directly aligning the feature maps between the two networks. Figure 1 As shown in (a) of the figure, the present invention adds similarity loss between the teacher and the student to enable the student network to further learn the inter-layer relationship of the teacher, so as to achieve further fine-grained constraints and improve the reconstruction ability of the model image. First, the similarity matrix of the module is generated. Taking the RCAN network as an example, as shown in the attached figure, Figure 1 In (a), The module consists of a header and 10 main blocks. The feature maps generated by the 10 main blocks are compared for correlation. Let X and Y represent the feature outputs of the intermediate layer, tr(*) represents the matrix trace, and T represents the matrix transpose. When a linear kernel is selected, the correlation between the two features X and Y can be obtained using the CKA formula:

[0078]

[0079]

[0080] Through the CKA method, the similarity relationship between different layers can be obtained, and the similarity relationship matrix A can be established. T1 :

[0081] A T1 =CKA(X (i) ,X (j) )

[0082] where X (i) ,X (j) Represents the feature output of different layers. Contains the features of 10 processed images, and a 10*10 matrix A can be obtained T1 .

[0083] Using the same formula, we can get the remaining 3 modules and The self-similarity matrix A T2 、A S1 and A S2 .

[0084] Then and The two modules generate their own similarity matrix A in their respective representation spaces T1 and A S2 , and The two modules generate a similarity matrix A in their own representation space T2 and A S1 Align two pairs and minimize the loss:

[0085]

[0086]

[0087] Minimize the loss function and By ensuring behavioral consistency between the student and teacher networks, the teacher's knowledge is transferred to the student module, achieving high-quality restoration of image detail textures. By minimizing the proposed similarity loss, the teacher's knowledge is transferred to the student module, ensuring the teacher's ability to extract image texture and detail, and transferring this ability to the student network. The use of relational features enables the student module to learn feature similarities within its own representational space, without having to directly imitate the teacher's complex representational space. This is because the teacher network generates a large amount of image texture, which is difficult for the student to directly imitate.

[0088] 5. Overall loss

[0089] In the cross-distillation stage, it consists of two parts: the reconstruction loss supervised by the urban landscape image and the behavioral similarity loss based on CKA. The reconstruction loss is composed of the super-resolution image output by the two paths and the high-resolution urban image. Norm composition, that is and The goal is to make the output of the entire network as close as possible to the real city image. The behavior similarity loss is and The purpose of the cross-distillation phase is to encourage students to imitate the intrinsic inter-layer similarity of teachers to achieve a more fine-grained supervision effect and preserve the texture details in the landscape. Therefore, the loss in the cross-distillation phase is:

[0090]

[0091]

[0092] In this stage, a total of 100 generations are trained.

[0093] In the fusion distillation stage, students are taken from two optimization paths without softening modules, and the two student modules are re-cascaded and combined. Further training is performed using the fusion loss. Unlike the previous stage, this stage uses the complete teacher Generate super-resolution images for supervision, namely:

[0094]

[0095] In this stage, the training was also performed for 100 generations.

[0096] The embodiments of the present invention for super-resolution dataset restoration tasks are as follows:

[0097] (1) Preparation

[0098] First, we need to prepare the DIV2K and Urban100 datasets required for the experiment. The DIV2K dataset contains 800 high-resolution images and their corresponding low-resolution counterparts for neural network training. The low-resolution images are downsampled from the high-resolution images to a quarter of their original size. Urban100 is a typical urban landscape dataset used in super-resolution tasks, containing 100 high-resolution images. Its textures are complex and cover a wide range of urban landscapes. This dataset serves as a test set for high-quality image super-resolution, demonstrating both high subjective visual quality and high objective metrics, to verify the effectiveness of the algorithm.

[0099] (2) Set hyperparameters, including the following hyperparameters as shown in Table 1:

[0100] Table 1

[0101] Hyperparameter name Initial learning rate epoch Batchsize Numerical 0.0001 200 16

[0102] (2) The DIV2K dataset was selected to train the network, and the network accuracy was tested after training. As shown in Table 2, when the super-resolution network was EDSR (Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1646–1654, 2016.) and RCAN, in the experiment with the test dataset Urban100, the EDSR network without distillation had a PSNR of 25.631 dB and an SSIM of 0.7707; after using the framework distillation proposed in this invention, the PSNR was 25.799 dB and the SSIM was 0.7766. The RCAN network without distillation had a PSNR of 26.340 dB and an SSIM of 0.7933; after using the framework distillation proposed in this invention, the PSNR was 26.519 dB and the SSIM was 0.7992. It can be concluded that after the distillation of the present invention, the objective indicators of the super-resolution network are significantly improved with the same number of parameters, which is conducive to the realization of high-quality application of downstream tasks related to urban landscape.

[0103] Table 2

[0104]

[0105]

[0106] (3) Image analysis: Figure 3 The leftmost image to be super-resolved is taken as an example of the analysis result. This image is a low-resolution image of a city landscape, characterized by regular texture structure and high repetition. The goal of the image super-resolution task is to obtain a larger super-resolution image, generally 2 times, 3 times, and 4 times. Compared with the teacher network that is difficult to deploy, the present invention compresses EDSR and RCAN by 32 times and 3 times respectively. Figure 3The rightmost column shows the 4x super-resolved images output by the RCAN model distilled by the present invention. The middle column shows the images output by the RCAN model directly trained without the present invention. The two models are of the same size and can be deployed on edge devices, but it can be observed that the results of the present invention are clearer and more consistent in restoring the urban landscape texture structure than the directly trained model, and the subjective visual experience is better. The algorithm of the present invention restores pleasant urban landscape images and can be quickly deployed on mobile phones; it can also help restore low-quality remote sensing satellite images, assist in the identification of urban landscapes and further geographic information processing.

[0107] Corresponding to the aforementioned embodiment of the super-resolution network compression method based on knowledge distillation, the present invention also provides an embodiment of an image super-resolution device based on a knowledge distillation compression model.

[0108] See also Figure 4 An embodiment of the present invention provides an image super-resolution device based on a knowledge distillation compression model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement an image super-resolution method based on a knowledge distillation compression model in the above embodiment.

[0109] An embodiment of an image super-resolution device based on a knowledge distillation compression model of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4 As shown in the figure, it is a hardware structure diagram of any device with data processing capability where an image super-resolution device based on knowledge distillation compression model of the present invention is located. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0110] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0111] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0112] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the image super-resolution method based on the knowledge distillation compression model in the above embodiment is implemented.

[0113] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0114] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. An image super-resolution method based on a knowledge distillation compression model, characterized in that: The method comprises the following steps: (1) Obtain a large trained teacher network and a small student network to be trained, wherein the teacher network is a public, pre-trained network; the student network is a network after the teacher network is reduced in depth, which will be used for actual deployment; the teacher network and the student network are divided into two modules in units of blocks according to the network depth, respectively, to obtain teacher module 1, teacher module 2, student module 1 and student module 2; and based on the softening module that integrates the semantic features of the previous layer and the features extracted from the current layer, the teacher module 1, the softening module and the student module are cascaded in turn, and the student module 1, the softening module and the teacher module are cascaded in turn to form two optimization paths for knowledge distillation; (2) Obtain low-definition images for the training set, where the training set is a public data set containing paired low-definition and high-definition images; input the low-definition images into two optimization paths for feature extraction; the network extracts the texture details and high-frequency features of the low-definition images as well as the structure of the image, and then generates a preliminary image result through the convolutional network. The loss term is calculated with the high-definition image to penalize the incorrect image restoration of the network; the loss term includes whether the structure is consistent, whether the color conforms to the statistical law, and whether the texture is natural. Finally, the parameters of the two paths are optimized based on the loss; (3) The two student modules after cross-distillation are taken out and connected to each other to form the final optimization path; the low-resolution image is input to the network, and the complete teacher network is used to generate a super-resolution image for supervision, the loss term is calculated and the parameters are updated. After the training is completed, the final compression model is obtained, and the obtained low-resolution image is input into the final compression model to obtain a super-resolution image.

2. The image super-resolution method based on the knowledge distillation compression model according to claim 1, characterized in that: The purpose of knowledge transfer is achieved by aligning the inter-layer relationship matrices of the teacher and the student. For the output features X and Y of different layers, the specific processing flow is as follows: A=CKA(X (i) ,X (j) ) Where tr(*) represents the trace of the matrix, T represents the transpose of the matrix; A T and A S Represent the inter-layer relationship matrices of the teacher network and the student network respectively; ||*||1 represents L1 regularization.

3. The image super-resolution method based on the knowledge distillation compression model according to claim 1, characterized in that: A softening module is added at the cross-cascade, and k learnable parameters are used to perform linear mapping on the feature matrix to achieve dimensional matching between the teacher and the student. A softening module is designed using channel separation and residual connection. The softening module uses residuals to retain the low-frequency image contour shape to prevent image edge blurring and network gradient disappearance. The softening module is used for further extraction of high-frequency textures through 3*3 convolutional layers. It is used to smoothly transfer knowledge between the teacher network and the student network. The softening module integrates the semantic features of the previous layer with the features extracted by the current layer, serving as a transition between the teacher and the student.

4. The image super-resolution method based on the knowledge distillation compression model according to claim 1, characterized in that: In the cross-distillation stage, the student network is optimized by minimizing the reconstruction loss and similarity loss; in the fusion distillation stage, the student network is optimized by minimizing the reconstruction loss with the teacher.

5. An image super-resolution device based on a knowledge distillation compression model, characterized in that: The device comprises: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors execute the image super-resolution method based on the knowledge distillation compression model as described in any one of claims 1 to 4.

6. A computer-readable storage medium for storing one or more computer programs, wherein the one or more computer programs include program codes, characterized in that: When the computer program is run on a computer, the program code is used to execute the image super-resolution method based on the knowledge distillation compression model according to any one of claims 1 to 4.

Images