Image super-resolution method and system based on deep feature correlation
Through the knowledge distillation technology of deep feature correlation knowledge, the problem of excessive parameters of image super-resolution algorithm in low-power devices is solved, and the effect of high-quality image reconstruction in low-power devices is achieved.
Patent Information
- Application Number
- CN202111074208.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-14
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-09-14
AI Technical Summary
Existing image super-resolution algorithms are difficult to deploy in low-power devices, mainly due to the large amount of calculations and parameters, which makes it impossible to effectively reconstruct high-quality high-resolution images.
Using the image super-resolution method based on deep feature correlation, the knowledge of deep feature correlation is extracted from the heavyweight teacher model through knowledge distillation technology, and supervised training is carried out in the lightweight student model, introducing distillation loss function and supervised loss function to compress model parameters and improve performance.
Under low parameters and calculations, the lightweight student model is able to reconstruct high-quality high-resolution images, solving the problem that heavyweight models cannot be deployed to low-power devices.
Smart Images

Figure CN113793265B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image super-resolution, and in particular to an image super-resolution method and system based on depth feature correlation. Background Art
[0002] In daily life, more and more low-power devices such as mobile phones and embedded terminals are widely used. People need to process low-resolution images in order to obtain high-resolution images with better visual effects on mobile devices. Therefore, the application of image super-resolution algorithms in low-power devices has received widespread attention.
[0003] The goal of image super-resolution technology is to estimate a high-resolution image from a low-resolution image. The degradation process of the image super-resolution problem is usually defined as:
[0004] L=SKI+n, (1)
[0005] Where L, I, n represent low-resolution image, high-resolution image and noise respectively, S and K represent the matrix form of downsampling matrix with scale factor and blur kernel respectively. Image super-resolution is an ill-posed problem because there are infinite pairs of blur kernel K and high-resolution image that can generate the same low-resolution image L. Traditional interpolation-based methods are simple and fast, but the quality of the restored high-resolution images is poor. In recent years, with the rapid development of deep learning technology, methods based on deep convolutional neural networks have greatly surpassed traditional interpolation-based methods in the reconstruction of high-resolution images, and related experiments have shown that the deeper the network with larger parameters, the more significantly it can improve the performance of image super-resolution algorithms, but it also brings about a significant increase in computing time and memory consumption. In real scenarios, these large models with huge computing and parameter amounts cannot be deployed on low-power devices such as mobile phones. In order to solve this problem, some model compression methods are needed to obtain some image super-resolution models with smaller parameters but better performance. Summary of the invention
[0006] The object of the present invention is to provide an image super-resolution method and system based on deep feature correlation, so as to estimate a high-resolution image through a student super-resolution model with small parameters and high performance.
[0007] To achieve the above object, the present invention provides the following solutions:
[0008] An image super-resolution method based on deep feature correlation, comprising:
[0009] Acquire a low-resolution image to be reconstructed;
[0010] Inputting the low-resolution image to be reconstructed into the trained student model for reconstruction to obtain a high-resolution image;
[0011] The training process of the student model includes:
[0012] A low-resolution training image is used as the input of the student model; a high-resolution training image is used as the output of the student model, and the overall loss function is used as the loss function. The student model is supervised and trained by the high-resolution image and the real image output by the trained teacher model using the loss function attenuation mechanism to obtain a trained student model; the trained teacher model includes ten residual group modules; the student model is obtained by compressing and replacing the teacher model; the overall loss function includes a distillation loss function and a supervision loss function.
[0013] Optionally, the method uses a low-resolution training image as the input of the student model; uses a high-resolution training image as the output of the student model, uses an overall loss function as the loss function, and uses a loss function attenuation mechanism to supervise the student model through high-resolution images and real images output by a trained teacher model to obtain a trained student model, specifically including:
[0014] Performing deep feature correlation calculation on the outputs of the residual group modules of different network depths of the student model and the trained teacher model to obtain a deep feature correlation matrix;
[0015] Determining a distillation loss function according to the deep feature correlation matrix;
[0016] Determine a supervision loss function based on the high-resolution image output by the trained teacher model and the real image;
[0017] A low-resolution training image is used as the input of the student model; a high-resolution training image is used as the output of the student model, and the student model is trained according to the distillation loss function and the supervision loss function by using a loss function attenuation mechanism to obtain a trained student model.
[0018] Optionally, before calculating the deep feature correlation of the outputs of the residual group modules of the student model and the trained teacher model with different network depths to obtain the deep feature correlation matrix, the method further includes:
[0019] The low-resolution training images are used as the input of the teacher model, the high-resolution training images are used as the output of the teacher model, and the super-resolution model is trained using a bicubic down-sampling paired data set to obtain a trained teacher model.
[0020] Optionally, the outputs of the residual group modules of the student model and the trained teacher model with different network depths are subjected to deep feature correlation calculation to obtain a deep feature correlation matrix, specifically including:
[0021] Performing feature mapping on the outputs of the residual group modules of the trained teacher model with different network depths to obtain the depth features of the teacher model after dimensionality reduction;
[0022] Normalizing and averaging the teacher model deep features after dimensionality reduction to obtain a teacher model deep feature correlation matrix;
[0023] The outputs of the residual group modules of different network depths of the student model are normalized and averaged to obtain a deep feature correlation matrix of the student model; the residual group module of the student model includes an inverted residual module; the deep feature correlation matrix includes a deep feature correlation matrix of a teacher model and a deep feature correlation matrix of a student model.
[0024] Optionally, determining the supervision loss function according to the high-resolution image output by the trained teacher model and the real image specifically includes:
[0025] The supervised loss function includes a first supervised loss function and a second supervised loss function;
[0026] Determining a first supervised loss function based on the high-resolution image output by the trained teacher model;
[0027] A second supervised loss function is determined according to the real image.
[0028] An image super-resolution system based on deep feature correlation, comprising:
[0029] An acquisition module, used for acquiring a low-resolution image to be reconstructed;
[0030] A reconstruction module, used for inputting the low-resolution image to be reconstructed into the trained student model for reconstruction to obtain a high-resolution image;
[0031] The reconstruction module includes a student model training submodule, which is used to use low-resolution training images as input of the student model; use high-resolution training images as output of the student model, use the overall loss function as the loss function, and utilize the loss function attenuation mechanism to supervise the student model through the high-resolution images and real images output by the trained teacher model to obtain a trained student model; the trained teacher model includes ten residual group modules; the student model is obtained by compressing and replacing the teacher model; the overall loss function includes a distillation loss function and a supervision loss function.
[0032] Optionally, the student model training submodule specifically includes:
[0033] A deep feature correlation matrix determination unit, used to perform deep feature correlation calculation on the outputs of the residual group modules of different network depths of the student model and the trained teacher model to obtain a deep feature correlation matrix;
[0034] A distillation loss function determining unit, used to determine a distillation loss function according to the deep feature correlation matrix;
[0035] A supervision loss function determination unit, used to determine a supervision loss function according to the high-resolution image output by the trained teacher model and the real image;
[0036] The student model training unit is used to use low-resolution training images as input of the student model; use high-resolution training images as output of the student model, train the student model according to the distillation loss function and the supervision loss function, and utilize the loss function attenuation mechanism to obtain a trained student model.
[0037] Optionally, the student model training submodule further includes:
[0038] The teacher model training unit is used to use low-resolution training images as input of the teacher model and high-resolution training images as output of the teacher model, and to train the super-resolution model using a paired data set of bicubic downsampling to obtain a trained teacher model.
[0039] Optionally, the deep feature correlation matrix determination unit specifically includes:
[0040] A feature mapping subunit is used to perform feature mapping on the outputs of the residual group modules of the trained teacher model with different network depths to obtain the depth features of the teacher model after dimensionality reduction;
[0041] A teacher model deep feature correlation matrix determination subunit is used to normalize and average the teacher model deep features after dimensionality reduction to obtain the teacher model deep feature correlation matrix;
[0042] The student model deep feature correlation matrix determination subunit is used to normalize and average the outputs of the residual group modules of different network depths of the student model to obtain the student model deep feature correlation matrix; the residual group module of the student model includes an inverted residual module; the deep feature correlation matrix includes the teacher model deep feature correlation matrix and the student model deep feature correlation matrix.
[0043] Optionally, the supervised loss function determining unit specifically includes: the supervised loss function includes a first supervised loss function and a second supervised loss function;
[0044] a first supervised loss function determination subunit, configured to determine a first supervised loss function according to the high-resolution image output by the trained teacher model;
[0045] The second supervised loss function determination subunit is used to determine the second supervised loss function according to the real image.
[0046] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0047] The present invention provides an image super-resolution method and system based on deep feature correlation. When training a student model, a low-resolution training image is used as the input of the student model; a high-resolution training image is used as the output of the student model, and an overall loss function is used as the loss function. The student model is supervised and trained by the high-resolution image and the real image output by the trained teacher model using the loss function attenuation mechanism to obtain a trained student model; the trained teacher model includes ten residual group modules; the student model is obtained by compressing and replacing the teacher model; the overall loss function includes a distillation loss function and a supervision loss function. By means of knowledge distillation, a distillation loss function is introduced to transfer the correlation information from the teacher model to the student module, so that the student model still has good performance when the number of parameters and the amount of calculation are low, and a high-quality high-resolution image is reconstructed. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0049] Figure 1 A schematic diagram of the process flow of the image super-resolution method based on deep feature correlation provided by the present invention;
[0050] Figure 2 A schematic diagram of the overall structure of a super-resolution network using knowledge distillation provided by the present invention;
[0051] Figure 3 A schematic diagram of the structure of the teacher model provided by the present invention;
[0052] Figure 4 A schematic diagram of the structure of the inverted residual module in the student model provided by the present invention;
[0053] Figure 5 This is a schematic diagram of the feature mapping module structure provided by the present invention. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] The object of the present invention is to provide an image super-resolution method and system based on deep feature correlation, so as to estimate a high-resolution image through a high-performance image super-resolution model with a small number of parameters.
[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0057] like Figure 1 and Figure 2 As shown, given a low-resolution image (LR), the present invention completes the task of transferring deep feature correlation knowledge from a heavyweight teacher model to a lightweight student model under the knowledge distillation framework, so that the lightweight student model can improve its performance under the supervision of the heavyweight teacher model and reconstruct a high-resolution image (SR) with better effect. The solid line in the flowchart represents the training process, and the dotted line represents the testing process. For the training process, the present invention uses a paired data set that has been bicubic downsampled for training. First, the teacher model is trained, and then the parameters are fixed. Then the student model is trained from scratch, and the deep feature correlation knowledge is extracted from the trained teacher model and passed to the student model for learning, so that the student model finally obtains better performance (solid line). When the student model gradually converges under the supervision of the teacher model, given a low-resolution image for testing, a high-resolution image can be reconstructed using the trained student model (dashed line). An image super-resolution method based on deep feature correlation provided by the present invention includes:
[0058] Obtain a low-resolution image to be reconstructed.
[0059] The low-resolution image to be reconstructed is input into the trained student model for reconstruction to obtain a high-resolution image.
[0060] The training process of the student model includes:
[0061] The low-resolution training image is used as the input of the student model; the high-resolution training image is used as the output of the student model, the overall loss function is used as the loss function, and the loss function attenuation mechanism is used to supervise the student model through the high-resolution image output by the trained teacher model and the real image to obtain the trained student model; the trained teacher model includes ten residual group modules; the student model is obtained by compressing and replacing the teacher model; the overall loss function includes a distillation loss function and a supervision loss function. That is, the low-resolution training image is used as the input of the student model; the high-resolution training image is used as the output of the student model, the overall loss function is used as the loss function, and the student model is trained by using the loss function attenuation mechanism to obtain the trained student model; the overall loss function includes a distillation loss function and a supervision loss function; the distillation loss function is determined according to the student model and the trained teacher model. The supervision loss function is determined according to the high-resolution image output by the trained teacher model and the real image; the trained teacher model includes ten residual group modules; the student model is obtained by compressing and replacing the teacher model.
[0062] In practical applications, the low-resolution training images are used as the input of the student model; the high-resolution training images are used as the output of the student model, the overall loss function is used as the loss function, and the loss function attenuation mechanism is used to supervise the student model through the high-resolution images and real images output by the trained teacher model to obtain the trained student model, which specifically includes:
[0063] The outputs of the residual group modules of the student model and the trained teacher model with different network depths are subjected to deep feature correlation calculation to obtain a deep feature correlation matrix.
[0064] A distillation loss function is determined according to the deep feature correlation matrix.
[0065] A supervision loss function is determined based on the high-resolution image output by the trained teacher model and the real image.
[0066] A low-resolution training image is used as the input of the student model; a high-resolution training image is used as the output of the student model, and the student model is trained according to the distillation loss function and the supervision loss function by using a loss function attenuation mechanism to obtain a trained student model.
[0067] In practical applications, before calculating the deep feature correlation of the outputs of the residual group modules of the student model and the trained teacher model with different network depths to obtain the deep feature correlation matrix, the method further includes:
[0068] The low-resolution training images are used as the input of the teacher model, the high-resolution training images are used as the output of the teacher model, and the super-resolution model is trained using a bicubic down-sampling paired data set to obtain a trained teacher model.
[0069] In practical applications, the outputs of the residual group modules of the student model and the trained teacher model with different network depths are subjected to deep feature correlation calculation to obtain a deep feature correlation matrix, which specifically includes:
[0070] The outputs of the residual group modules of the trained teacher model with different network depths are feature mapped to obtain the depth features of the teacher model after dimensionality reduction.
[0071] The teacher model deep features after dimensionality reduction are normalized and averaged to obtain the teacher model deep feature correlation matrix.
[0072] The outputs of the residual group modules of different network depths of the student model are normalized and averaged to obtain a deep feature correlation matrix of the student model; the residual group module of the student model includes an inverted residual module; the deep feature correlation matrix includes a deep feature correlation matrix of a teacher model and a deep feature correlation matrix of a student model.
[0073] In practical applications, determining the supervision loss function based on the high-resolution image output by the trained teacher model and the real image specifically includes:
[0074] The supervised loss function includes a first supervised loss function and a second supervised loss function.
[0075] A first supervised loss function is determined based on the high-resolution image output by the trained teacher model.
[0076] A second supervised loss function is determined according to the real image.
[0077] The present invention also provides a more specific implementation of an image super-resolution method based on deep feature correlation, including the following contents:
[0078] Select the teacher model and train it
[0079] In order to build the framework of knowledge distillation, we first select a Figure 3The heavyweight super-resolution model RCAN (residual channel attention network) shown in the figure is used as the teacher model. Its construction form is nested residual. The model consists of ten residual group modules, each of which consists of twenty residual modules. Each residual module consists of four layers of standard convolution. Each convolution layer is connected to an activation layer. It has high performance in image super-resolution tasks. The form of its network is shown in formula (2).
[0080] I SR =T(I LR ) (2)
[0081] Where T refers to the teacher model, I LR and I SR They refer to the low-resolution image and the high-resolution image output by the teacher model, respectively.
[0082] After the teacher model is selected, the teacher model is trained. First, the low-resolution image (LR) is input into the teacher model, and a layer of convolution is used to extract the shallow feature F0, and then the high-level feature F is obtained by nesting the residual module. DF , at the tail of the nested residual structure, the shallow feature F0 and the high-level feature F DF Superposition is performed, and then upsampling is performed through the upsampling module to obtain the feature F up Finally, a high-resolution image (SR) is obtained through the reconstruction module. The loss function formula (3) is used to constrain the high-resolution image generated by the teacher model, and then the parameters of the trained teacher model are fixed. In the subsequent training process of the student model, back propagation is not performed, but deep features of different network depths are provided to calculate the correlation knowledge and pass it to the student model for learning.
[0083]
[0084] Where T refers to the teacher model, and are low-resolution images and true high-resolution images, L cons_T is the loss function of the teacher model training process.
[0085] Building a student model
[0086] The student model is obtained by compressing the number of residual modules and channels in the teacher model. Specifically, the number of residual modules in the teacher model is reduced from 20 to 2, and the number of feature channels in the teacher model is reduced from 64 to 16. The standard convolution in the residual module of the teacher model is replaced by an optimized lightweight inverted residual module. Figure 4As shown, a lightweight student model is finally constructed.
[0087] The optimized lightweight inverted residual module consists of four separate layers. The first layer is called the expansion layer, which uses 1x1 convolution to map features from low-dimensional space to high-dimensional space. The second layer is called the deep convolution layer, which performs lightweight filtering on each input channel through group convolution. The third layer is the ReLU activation layer, which performs nonlinear mapping of features. The fourth layer is called the projection layer, which also uses 1x1 convolution to map feature maps from high-dimensional space to low-dimensional space, and constructs new features by calculating linear combinations of input features. The purpose of using the inverted residual module is to reduce the amount of calculation of the student model and speed up the reasoning speed of the student model.
[0088] Since the inverted residual module introduces group convolution operations and 1x1 convolution, it can reduce the number of parameters and calculations of the student model without reducing the performance too much. The inverted residual module can reduce the calculation amount of the standard convolution by 8 to 9 times. It is expressed by the following formula:
[0089]
[0090] in, is the inverted residual module, F input Refers to the input high-dimensional features, F output Refers to the output high-dimensional features of the module.
[0091] Training the student model
[0092] Deep Feature Maps in Teacher Model
[0093] Under the framework of knowledge distillation, the student model is trained from scratch.
[0094] First, the low-resolution image (LR) is input into the trained teacher model and the student model at the same time. Then, the outputs of the residual groups of different network depths in the teacher model and the student model, such as the first, fifth, and tenth, are used as deep features. Since the number of channels in the model is compressed when the student model is constructed, the dimensions of the deep features extracted from the teacher model and the student model are inconsistent. Therefore, the following is adopted: Figure 5 The feature mapping module shown is the feature mapping unit. Drawing on the minimum error method of PCA, the three deep features of the teacher model are processed using the feature mapping module. The feature mapping module consists of two layers of 1x1 convolutional layers. The first layer of convolution is used to map the deep features of the teacher model from high dimension to low dimension, and then the second layer of convolution is used to map the low-dimensional deep features to output high-dimensional features, so that the low-dimensional deep features F in the middle of the feature mapping module are reductIt can retain the main feature information to reconstruct the output high-dimensional features that are similar to the input high-dimensional features as much as possible. The feature mapping module is shown as follows:
[0095]
[0096] in, is the feature mapping module, F input Refers to the input high-dimensional features, F output Refers to the output high-dimensional features of the module.
[0097] During the distillation training process, since the dimensions of the deep features in the teacher model and the student model are inconsistent, a feature mapping module is needed to reduce the dimension of the deep features in the teacher model. During the training process of the feature mapping module, the mean square error cost function formula (6) is used to constrain the output high-dimensional features of the feature mapping module:
[0098]
[0099] Among them, F input Refers to the input high-dimensional features, F output Refers to the output high-dimensional features of the module, N is the number of samples in a batch, and i is the i-th sample in a batch.
[0100] The low-dimensional deep feature F in the feature mapping module is obtained reduct Finally, the deep feature correlation calculation module is used to mine the correlation knowledge of deep features. This module will extract the feature correlation matrix to represent the intrinsic correlation between different features. The specific content of calculating the deep feature correlation matrix from the deep features is as follows:
[0101] Assume that the dimension of the low-dimensional deep feature after feature mapping of the given teacher model is F∈R C×W×H , first compress the last two dimensions of the deep feature F into one dimension, compressed into a dimension of A two-dimensional tensor of , where C is the number of channels, WH is the product of length and width, and R is a real number.
[0102] In order to effectively transfer the rich knowledge of deep features in the teacher model during the knowledge distillation process, the deep features generated by the teacher model and the student model are confined to a solution space for calculation. Each column of is normalized to limit the size of the solution space. Normalization means that the data minus the minimum value of the current dimension is divided by the maximum value of the current dimension minus the minimum value of the current dimension. This can limit the data to the range of 0 to 1. After the normalization operation, in order to compress the calculation amount of the deep feature correlation calculation, the deep features after the normalization operation are first averaged in the channel dimension, as shown in formula (7). The processed deep feature dimension is Then the deep feature correlation matrix D is calculated using formula (8).
[0103]
[0104] in refers to The transpose of is the compressed depth feature, is the depth feature after taking the second dimension, WH is the value of the first dimension of the depth feature multiplied by the second dimension, and [i, j] is the feature point with coordinates i, j in the depth feature.
[0105] During the student model training process, both the teacher model and the student model calculate the deep feature correlation matrix. The distillation loss function formula (9) is introduced to supervise the student model to generate the student model deep feature correlation matrix D S Try to match the teacher model deep feature correlation matrix D generated by the teacher model T Consistent, so that the student model can learn the correlation of deep features in the teacher model.
[0106]
[0107] Among them, L fd is the distillation loss function, is the teacher model correlation matrix, is the student model correlation matrix.
[0108] In addition to introducing distillation loss for training, the high-resolution image (SR) output by the teacher model and the real high-resolution image (HR) are used to supervise the learning of the student model, and formula (10), i.e. the first supervision loss function, and formula (11), i.e. the second supervision loss function, are used to further constrain the output high-resolution image (SR) of the student model.
[0109]
[0110] Among them, T refers to the teacher model, S refers to the student model, refers to the i-th low-resolution image, N refers to the number of images in a batch, and L sr_cons is the first supervised loss function, is the output of the teacher model, is the output of the student model.
[0111]
[0112] in, refers to the i-th real high-resolution image.
[0113] Finally, the overall loss function of the student model is shown in formula (12).
[0114] L(θ)=aL cons_s +βL sr _ cons +γL fd (12)
[0115] Among them, α, β, and γ are weight parameters used to balance the effects of different loss functions.
[0116] In terms of the selection of weight parameters, in the early stages of student network training, the distillation loss function can help the student model quickly learn the effective knowledge of the teacher model, thereby accelerating the network convergence of the student model. However, the distillation loss function will inhibit the further learning of the student model in the later stages of student model training. The weight attenuation mechanism of the loss function is used to gradually reduce the effect of the distillation loss during the training process, further improving the effect of knowledge distillation:
[0117]
[0118] Where σ is a decay constant coefficient, n e It refers to the current epoch number in the entire training process. n is a variable parameter used to determine how many epochs are decayed.
[0119] The present invention also provides an image super-resolution system based on deep feature correlation, comprising:
[0120] The acquisition module is used to acquire the low-resolution image to be reconstructed.
[0121] The reconstruction module is used to input the low-resolution image to be reconstructed into the trained student model for reconstruction to obtain a high-resolution image.
[0122] The reconstruction module includes a student model training submodule, which is used to use low-resolution training images as input of the student model; use high-resolution training images as output of the student model, use the overall loss function as the loss function, and utilize the loss function attenuation mechanism to supervise the student model through the high-resolution images and real images output by the trained teacher model to obtain a trained student model; the trained teacher model includes ten residual group modules; the student model is obtained by compressing and replacing the teacher model; the overall loss function includes a distillation loss function and a supervision loss function.
[0123] In practical applications, the student model training submodule specifically includes:
[0124] The deep feature correlation matrix determination unit is used to calculate the deep feature correlation of the outputs of the residual group modules of the student model and the trained teacher model with different network depths to obtain a deep feature correlation matrix.
[0125] A distillation loss function determining unit is used to determine a distillation loss function according to the deep feature correlation matrix.
[0126] A supervision loss function determination unit is used to determine a supervision loss function based on the high-resolution image output by the trained teacher model and the real image.
[0127] The student model training unit is used to use low-resolution training images as input of the student model; use high-resolution training images as output of the student model, train the student model according to the distillation loss function and the supervision loss function, and utilize the loss function attenuation mechanism to obtain a trained student model.
[0128] In practical applications, the student model training submodule further includes:
[0129] The teacher model training unit is used to use low-resolution training images as input of the teacher model and high-resolution training images as output of the teacher model, and to train the super-resolution model using a paired data set of bicubic downsampling to obtain a trained teacher model.
[0130] In practical applications, the deep feature correlation matrix determination unit specifically includes:
[0131] The feature mapping subunit is used to perform feature mapping on the output of the residual group module of different network depths of the trained teacher model to obtain the depth features of the teacher model after dimensionality reduction.
[0132] The teacher model deep feature correlation matrix determination subunit is used to normalize and average the teacher model deep features after dimensionality reduction to obtain the teacher model deep feature correlation matrix.
[0133] The student model deep feature correlation matrix determination subunit is used to normalize and average the outputs of the residual group modules of different network depths of the student model to obtain the student model deep feature correlation matrix; the residual group module of the student model includes an inverted residual module; the deep feature correlation matrix includes the teacher model deep feature correlation matrix and the student model deep feature correlation matrix.
[0134] In practical applications, the supervised loss function determination unit specifically includes: the supervised loss function includes a first supervised loss function and a second supervised loss function.
[0135] The first supervised loss function determination subunit is used to determine the first supervised loss function according to the high-resolution image output by the trained teacher model.
[0136] The second supervised loss function determination subunit is used to determine the second supervised loss function according to the real image.
[0137] In order to solve the problem of excessive model parameters, the present invention introduces an inverted residual module to replace the standard convolution layer, and uses knowledge distillation to transfer the correlation knowledge from the heavyweight super-resolution model to the lightweight super-resolution model by introducing distillation loss under the framework of knowledge distillation, so that the lightweight super-resolution model still has good performance when the number of parameters and the amount of calculation are low, thereby improving the performance of the lightweight super-resolution model. This solves the problem that the heavyweight super-resolution model cannot be deployed on low-power embedded devices.
[0138] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0139] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. An image super-resolution method based on deep feature correlation, characterized in that: include: Acquire a low-resolution image to be reconstructed; Inputting the low-resolution image to be reconstructed into the trained student model for reconstruction to obtain a high-resolution image; The training process of the student model includes: Using low-resolution training images as input to the student model; The high-resolution training image is used as the output of the student model, the overall loss function is used as the loss function, and the loss function attenuation mechanism is used to supervise the student model through the high-resolution image and the real image output by the trained teacher model to obtain a trained student model; the trained teacher model includes ten residual group modules; The student model is obtained by compressing and replacing the teacher model; the overall loss function includes a distillation loss function and a supervision loss function. Specifically, the student model is obtained by compressing the number of residual modules and the number of channels in the teacher model, specifically, the number of residual modules in the teacher model is reduced from 20 to 2, the number of feature channels in the teacher model is reduced from 64 to 16, and the standard convolution in the residual module of the teacher model is replaced by an optimized lightweight inverted residual module. The optimized lightweight inverted residual module consists of four separate layers. The first layer is an expansion layer, which uses 1x1 convolution to map features from a low-dimensional space to a high-dimensional space; the second layer is a deep convolution layer, which performs lightweight filtering on each input channel through group convolution; the third layer is a ReLU activation layer, which performs nonlinear mapping of features, and the fourth layer is a projection layer, which uses 1x1 convolution to map feature maps from a high-dimensional space to a low-dimensional space.
2. The image super-resolution method based on deep feature correlation according to claim 1, characterized in that: The method uses a low-resolution training image as the input of the student model; uses a high-resolution training image as the output of the student model, uses an overall loss function as the loss function, and uses a loss function attenuation mechanism to supervise the student model through the high-resolution image and the real image output by the trained teacher model to obtain a trained student model, specifically including: Performing deep feature correlation calculation on the outputs of the residual group modules of different network depths of the student model and the trained teacher model to obtain a deep feature correlation matrix; Determining a distillation loss function according to the deep feature correlation matrix; Determine a supervision loss function based on the high-resolution image output by the trained teacher model and the real image; A low-resolution training image is used as the input of the student model; a high-resolution training image is used as the output of the student model, and the student model is trained according to the distillation loss function and the supervision loss function by using a loss function attenuation mechanism to obtain a trained student model.
3. The image super-resolution method based on deep feature correlation according to claim 2, characterized in that: Before calculating the deep feature correlation of the outputs of the residual group modules of the student model and the trained teacher model with different network depths to obtain the deep feature correlation matrix, the method further includes: The low-resolution training images are used as the input of the teacher model, the high-resolution training images are used as the output of the teacher model, and the super-resolution model is trained using a bicubic down-sampling paired data set to obtain a trained teacher model.
4. The image super-resolution method based on deep feature correlation according to claim 2, characterized in that: The outputs of the residual group modules of the student model and the trained teacher model with different network depths are subjected to deep feature correlation calculation to obtain a deep feature correlation matrix, specifically including: Performing feature mapping on the outputs of the residual group modules of the trained teacher model with different network depths to obtain the depth features of the teacher model after dimensionality reduction; Normalizing and averaging the teacher model deep features after dimensionality reduction to obtain a teacher model deep feature correlation matrix; The outputs of the residual group modules of different network depths of the student model are normalized and averaged to obtain a deep feature correlation matrix of the student model; the residual group module of the student model includes an inverted residual module; the deep feature correlation matrix includes a deep feature correlation matrix of a teacher model and a deep feature correlation matrix of a student model.
5. The image super-resolution method based on deep feature correlation according to claim 2, characterized in that: Determining the supervision loss function according to the high-resolution image output by the trained teacher model and the real image specifically includes: The supervised loss function includes a first supervised loss function and a second supervised loss function; Determining a first supervised loss function based on the high-resolution image output by the trained teacher model; A second supervised loss function is determined according to the real image.
6. An image super-resolution system based on deep feature correlation, characterized in that: include: An acquisition module, used for acquiring a low-resolution image to be reconstructed; A reconstruction module, used for inputting the low-resolution image to be reconstructed into the trained student model for reconstruction to obtain a high-resolution image; The reconstruction module includes a student model training submodule, which is used to use low-resolution training images as input of the student model; use high-resolution training images as output of the student model, use the overall loss function as the loss function, and use the loss function attenuation mechanism to supervise the student model through the high-resolution images and real images output by the trained teacher model to obtain a trained student model; the trained teacher model includes ten residual group modules; the student model is obtained by compressing and replacing the teacher model; the overall loss function includes a distillation loss function and a supervision loss function. Specifically, the student model is obtained by compressing the residual module in the teacher model. The optimization is achieved by reducing the number of residual modules in the teacher model from 20 to 2, the number of feature channels in the teacher model from 64 to 16, and replacing the standard convolution in the residual module of the teacher model with an optimized lightweight inverted residual module. The optimized lightweight inverted residual module consists of four separate layers. The first layer is called the expansion layer, which uses 1x1 convolution to map features from low-dimensional space to high-dimensional space; the second layer is called the deep convolution layer, which performs lightweight filtering on each input channel through group convolution; the third layer is the ReLU activation layer, which performs nonlinear mapping of features; the fourth layer is called the projection layer, which uses 1x1 convolution to map feature maps from high-dimensional space to low-dimensional space.
7. The image super-resolution system based on deep feature correlation according to claim 6, characterized in that: The student model training submodule specifically includes: A deep feature correlation matrix determination unit, used to perform deep feature correlation calculation on the outputs of the residual group modules of different network depths of the student model and the trained teacher model to obtain a deep feature correlation matrix; A distillation loss function determining unit, used to determine a distillation loss function according to the deep feature correlation matrix; A supervision loss function determination unit, used to determine a supervision loss function according to the high-resolution image output by the trained teacher model and the real image; The student model training unit is used to use low-resolution training images as input of the student model; use high-resolution training images as output of the student model, train the student model according to the distillation loss function and the supervision loss function, and utilize the loss function attenuation mechanism to obtain a trained student model.
8. The image super-resolution system based on deep feature correlation according to claim 7, characterized in that: The student model training submodule further includes: The teacher model training unit is used to use low-resolution training images as input of the teacher model and high-resolution training images as output of the teacher model, and to train the super-resolution model using a paired data set of bicubic downsampling to obtain a trained teacher model.
9. The image super-resolution system based on deep feature correlation according to claim 7, characterized in that: The deep feature correlation matrix determination unit specifically includes: A feature mapping subunit is used to perform feature mapping on the outputs of the residual group modules of the trained teacher model with different network depths to obtain the depth features of the teacher model after dimensionality reduction; A teacher model deep feature correlation matrix determination subunit is used to normalize and average the teacher model deep features after dimensionality reduction to obtain the teacher model deep feature correlation matrix; The student model deep feature correlation matrix determination subunit is used to normalize and average the outputs of the residual group modules of different network depths of the student model to obtain the student model deep feature correlation matrix; the residual group module of the student model includes an inverted residual module; the deep feature correlation matrix includes the teacher model deep feature correlation matrix and the student model deep feature correlation matrix.
10. The image super-resolution system based on deep feature correlation according to claim 7, characterized in that: The supervisory loss function determination unit specifically includes: the supervisory loss function includes a first supervisory loss function and a second supervisory loss function; a first supervised loss function determination subunit, configured to determine a first supervised loss function according to the high-resolution image output by the trained teacher model; The second supervised loss function determination subunit is used to determine the second supervised loss function according to the real image.
Citation Information
Patent Citations
Generation method and reconstruction method of image super-resolution reconstruction model and electronic device
CN112200722A
