Super-resolution reconstruction model training method, image super-resolution reconstruction method and electronic equipment

Abstract

The embodiment of the invention provides a super-resolution reconstruction model training method, an image super-resolution reconstruction method and electronic equipment, and relates to the technical field of super-resolution reconstruction. Performing up-sampling and wavelet decomposition on the low-resolution image to obtain a first low-frequency sub-band image and a first high-frequency sub-band image, and inputting the first low-frequency sub-band image and the first high-frequency sub-band image into the trained teacher super-division reconstruction network and the trained student super-division reconstruction network respectively to obtain a trained teacher super-division reconstruction network and a trained student super-division reconstruction network; and calculating loss information according to the output of the teacher super-division reconstruction network and the student super-division reconstruction network, carrying out iterative optimization on the student super-division reconstruction network according to the loss information until a preset condition is met, and taking the optimized student super-division reconstruction network as a super-division reconstruction model. Therefore, the network layer number and the network parameter quantity of the super-division reconstruction model can be reduced, and the calculation overhead of super-division reconstruction is further reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of super-resolution reconstruction, and in particular to a super-resolution reconstruction model training method, an image super-resolution reconstruction method and an electronic device. Background Art

[0002] Image super-resolution reconstruction refers to the technology of restoring low-resolution images to high-resolution images by using image processing, deep learning and other methods.

[0003] In the prior art, the image super-resolution reconstruction method based on deep learning basically uses bicubic interpolation and other methods to first upsample the low-resolution image to a high-resolution image, then extract features from the high-resolution image, and finally reconstruct the super-resolution image. However, as the number of network layers increases, the number of network parameters of this type of image super-resolution reconstruction method based on deep learning increases, and the computational overhead also continues to increase. Summary of the invention

[0004] In view of this, an object of an embodiment of the present invention is to provide a super-resolution reconstruction model training method, an image super-resolution reconstruction method and an electronic device to at least partially improve the above-mentioned problem.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a super-resolution reconstruction model training method, the method comprising:

[0007] Upsampling the low-resolution image and performing wavelet decomposition to obtain a first low-frequency sub-band image and a first high-frequency sub-band image;

[0008] Inputting the first low-frequency sub-band image and the first high-frequency sub-band image into a trained teacher super-resolution reconstruction network to obtain an L-layer teacher feature map in the feature extraction process and a teacher super-resolution image after super-resolution reconstruction;

[0009] Inputting the first low-frequency sub-band image and the first high-frequency sub-band image into a student super-resolution reconstruction network to obtain a processed second low-frequency sub-band image and a processed second high-frequency sub-band image, an L-layer student feature map in the feature extraction process, and a student super-resolution image after super-resolution reconstruction; wherein the architecture of the student super-resolution reconstruction network is the same as that of the teacher super-resolution reconstruction network, and the parameters of the student super-resolution reconstruction network are smaller than the parameters of the teacher super-resolution reconstruction network;

[0010] Calculating loss information according to the low-resolution image, each of the teacher feature maps, the teacher super-resolution image, each of the student feature maps, the student super-resolution image, the first low-frequency sub-band image, the first high-frequency sub-band image, the second low-frequency sub-band image, and the second high-frequency sub-band image;

[0011] The student super-resolution reconstruction network is iteratively optimized according to the loss information until a preset condition is met, and the optimized student super-resolution reconstruction network is used as a super-resolution reconstruction model.

[0012] Optionally, the student super-resolution reconstruction network includes a convolutional network and a plurality of multi-scale aggregation modules, and the first low-frequency sub-band image and the first high-frequency sub-band image are input into the student super-resolution reconstruction network to obtain a processed second low-frequency sub-band image and a second high-frequency sub-band image, an L-layer student feature map in a feature extraction process, and a student super-resolution image after super-resolution reconstruction, including:

[0013] Inputting the first low-frequency sub-band image and the first high-frequency sub-band image into the convolutional network to obtain shallow features;

[0014] Input the shallow features into each of the multi-scale aggregation modules in sequence to obtain the L-layer student feature map in the feature extraction process and the final deep features;

[0015] Fusing the shallow features with the deep features to obtain fused wavelet coefficients; the wavelet coefficients include the fused second low-frequency sub-band image and the second high-frequency sub-band image;

[0016] The wavelet coefficients are subjected to inverse wavelet reconstruction to obtain a student super-resolution image.

[0017] Optionally, the shallow features are sequentially input into each of the multi-scale aggregation modules to obtain L-layer student feature maps in the feature extraction process and the final deep features, including:

[0018] Inputting the shallow features into the first multi-scale aggregation module to obtain a first deep feature and a plurality of student feature maps;

[0019] The input features of the previous multi-scale aggregation module and the first deep features are sequentially input into the next multi-scale aggregation module until the last multi-scale aggregation module to generate the final deep features.

[0020] Optionally, the multi-scale aggregation module includes a first fusion layer, a grouped convolution layer, N convolution layers, a channel shuffle layer and a second fusion layer, and the shallow features are input into the first multi-scale aggregation module to obtain the first deep features and multiple student feature maps, including:

[0021] Inputting the shallow layer features into the first fusion layer to obtain a first fusion feature;

[0022] Inputting the first fusion feature into the group convolution layer to obtain N first group features;

[0023] Inputting N first grouping features into N convolutional layers in sequence, and using the output of the previous convolutional layer as part of the input of the current convolutional layer, to obtain N second grouping features with an increased receptive field;

[0024] Inputting the N second grouping features into the channel shuffling layer to obtain N third grouping features with rearranged channels;

[0025] Inputting the N third grouping features into the second fusion layer to obtain a second fusion feature;

[0026] Fusing the shallow feature with the second fused feature to obtain a first deep feature;

[0027] The first fusion feature, N of the second grouping features, N of the third grouping features, the second fusion feature and the first deep feature are recorded as a student feature map.

[0028] Optionally, calculating the loss information according to the low-resolution image, each of the teacher feature maps, the teacher super-resolution image, each of the student feature maps, the student super-resolution image, the first low-frequency sub-band image, the first high-frequency sub-band image, the second low-frequency sub-band image, and the second high-frequency sub-band image includes:

[0029] Based on the teacher supervision loss function, according to the teacher super-resolution image and the student super-resolution image, the teacher supervision loss information of the teacher super-resolution reconstruction network supervising the student super-resolution reconstruction network is calculated;

[0030] Based on the data supervision loss function, according to the student super-resolution image and the low-resolution image, the data supervision loss information of the low-resolution image on the student super-resolution reconstruction network is calculated;

[0031] Calculating feature distillation loss information according to each of the teacher feature graphs and each of the student feature graphs;

[0032] Based on the wavelet prediction loss function, the wavelet prediction loss information of the student super-resolution image is calculated according to the first low-frequency sub-band image, the first high-frequency sub-band image, the second low-frequency sub-band image and the second high-frequency sub-band image;

[0033] Based on the total loss function, the loss information is calculated according to the teacher supervision loss information, the data supervision loss information, the feature distillation loss information and the wavelet prediction loss information.

[0034] Optionally, calculating feature distillation loss information according to each of the teacher feature graphs and each of the student feature graphs includes:

[0035] Each of the teacher characteristic graphs and each of the student characteristic graphs are converted into vectors and normalized to obtain a corresponding teacher normalized column vector and a student normalized column vector;

[0036] Calculate the characteristic matrix of each teacher normalized column vector and each student normalized column vector to obtain the corresponding teacher characteristic matrix and student characteristic matrix respectively;

[0037] Based on the feature distillation loss function, the feature distillation loss information is calculated according to each of the teacher feature matrices and each of the student feature matrices.

[0038] Optionally, the feature distillation loss function is:

[0039]

[0040] Wherein, |M| represents the number of elements of each teacher feature matrix and each student feature matrix, L represents the number of the student feature graphs or the number of the teacher feature graphs, represents the student feature matrix extracted from the student feature map of the lth layer, represents the student feature matrix extracted from the teacher feature map of the lth layer, and ||·||1 represents the l1 norm.

[0041] Optionally, the teacher supervision loss function is:

[0042]

[0043] in, represents the teacher super-resolution image, represents the student super-resolution image;

[0044] The data supervision loss function is:

[0045]

[0046] in, represents the student super-resolution image, representing the low-resolution image;

[0047] The wavelet prediction loss function is:

[0048]

[0049] in, denote the first low-frequency sub-band image and the first high-frequency sub-band image respectively, f denotes the student super-resolution reconstruction network, represent the second low-frequency sub-band image and the second high-frequency sub-band image respectively, and N represents the number of training samples in each batch;

[0050] The total loss function is:

[0051] L total =α1L TS +α2L DS +α3L kd +α4L wav

[0052] Among them, α1, α2, α3, α4 all represent loss weights, L kd represents the feature distillation loss function.

[0053] In a second aspect, an embodiment of the present invention provides an image super-resolution reconstruction method, the method comprising:

[0054] Obtain a low-resolution image to be super-reconstructed;

[0055] The low-resolution image is input into a super-resolution reconstruction model trained by any of the methods described above to obtain a super-resolution image after super-resolution reconstruction.

[0056] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-described methods when executing the program.

[0057] The embodiments of the present invention provide a super-resolution reconstruction model training method, an image super-resolution reconstruction method and an electronic device, which fully utilize the information in the image space domain and frequency domain and the relationship between them through wavelet and knowledge distillation, explore the multi-scale characteristics of the image after wavelet decomposition, obtain the deep potential features of the image, and realize feature prior knowledge transfer learning, thereby reducing the storage overhead of the network, improving the computing efficiency of the network, and enhancing the performance of the image super-resolution reconstruction model; through the feature extraction of the multi-scale aggregation module, richer image multi-scale features can be extracted with fewer parameters, increasing the receptive field of the network, and further improving the performance of image super-resolution reconstruction.

[0058] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a process flow of a super-resolution reconstruction model training method provided by an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of a flow chart of step S230 provided in an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of a process of step S2321 provided in an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of the structure of a multi-scale aggregation module provided by an embodiment of the present invention;

[0065] Figure 6 A schematic diagram of a flow chart of an image super-resolution reconstruction method provided by an embodiment of the present invention.

[0066] Icon: 100 - electronic device; 101 - memory; 102 - communication interface; 103 - processor; 104 - bus. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0068] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0070] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0071] In the prior art, the image super-resolution reconstruction method based on deep learning basically adopts bicubic interpolation and other methods to first upsample the low-resolution image to a high-resolution image, then extract features from the high-resolution image, and finally reconstruct the super-resolution image. As the number of network layers increases, the number of network parameters of this type of image super-resolution reconstruction method based on deep learning increases, and the computational overhead also increases. This is not suitable for super-resolution image reconstruction scenarios with high real-time requirements.

[0072] Based on the above situation, the embodiments of the present invention provide a super-resolution reconstruction model training method, an image super-resolution reconstruction method and an electronic device, which fully utilizes the information in the image space domain and frequency domain and the relationship between them through wavelet and knowledge distillation, explores the multi-scale characteristics of the image after wavelet decomposition, obtains the deep potential features of the image, and realizes feature prior knowledge transfer learning, reduces the storage overhead of the network, improves the computing efficiency of the network, and improves the performance of the image super-resolution reconstruction model.

[0073] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0074] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.

[0075] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0076] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0077] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0078] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0079] The following is an exemplary description of the super-resolution reconstruction model training method provided by the present invention. Specifically, Figure 2 A flowchart of a super-resolution reconstruction model training method provided by an embodiment of the present invention is shown in FIG. Figure 2 , the execution subject of this method can be the above Figure 1 The electronic device 100 shown in FIG. 1 includes: Figure 2 The following steps are described:

[0080] S210: Up-sampling the low-resolution image and performing wavelet decomposition to obtain a first low-frequency sub-band image and a first high-frequency sub-band image.

[0081] During the training process, each training batch may have multiple training sample data, that is, multiple low-resolution images. For ease of explanation, in this embodiment, the processing process of a low-resolution image is explained.

[0082] Optionally, in this embodiment, the low-resolution image I is first LR Bicubic interpolation upsampling is performed, and then the upsampled image is subjected to wavelet transform to obtain 4 sub-band images, including 1 first low-frequency sub-band image and 3 first high-frequency sub-band images. The 4 sub-band images are used as the input of the convolution layer together, and the parameters are shared, which is equivalent to implementing 4-channel image processing.

[0083] For example, using the ChinaSet dataset as training data, the input low-resolution image I LR The size is 128×128, and it is upsampled by bicubic interpolation. The upsampled image is recorded as Its size is 512×512. The number of layers for wavelet decomposition is set to 1. Perform wavelet decomposition W D After that, a first low-frequency subband image I will be obtained LF and 3 first high frequency sub-band images I HF .

[0084] S220: Input the first low-frequency sub-band image and the first high-frequency sub-band image into the trained teacher super-resolution reconstruction network to obtain the L-layer teacher feature map in the feature extraction process and the teacher super-resolution image after super-resolution reconstruction.

[0085] The first low-frequency sub-band image and the first high-frequency sub-band image are input into the trained teacher super-resolution reconstruction network. The teacher super-resolution reconstruction network extracts features from each sub-band image. During the feature extraction process, multiple feature maps are obtained, and finally a super-resolution image is output. The multiple feature maps are recorded as L-layer teacher feature maps, and the super-resolution image is recorded as a teacher super-resolution image.

[0086] S230: Input the first low-frequency subband image and the first high-frequency subband image into the student super-resolution reconstruction network to obtain the processed second low-frequency subband image and the second high-frequency subband image, the L-layer student feature map in the feature extraction process, and the student super-resolution image after super-resolution reconstruction. The architecture of the student super-resolution reconstruction network is the same as that of the teacher super-resolution reconstruction network, and the parameters of the student super-resolution reconstruction network are smaller than the parameters of the teacher super-resolution reconstruction network.

[0087] Since the architecture of the student super-resolution reconstruction network is the same as that of the teacher super-resolution reconstruction network, the principle is the same as step S220, and L-layer student feature maps and student super-resolution images after super-resolution reconstruction will be obtained. In the process of processing the first low-frequency sub-band image and the first high-frequency sub-band image, a processed second low-frequency sub-band image and a processed second high-frequency sub-band image will be obtained.

[0088] S240: Calculate loss information based on the low-resolution image, each teacher feature map, the teacher super-resolution image, each student feature map, the student super-resolution image, the first low-frequency sub-band image, the first high-frequency sub-band image, the second low-frequency sub-band image and the second high-frequency sub-band image.

[0089] After the first low-frequency sub-band image and the first high-frequency sub-band image are processed by the student super-resolution reconstruction network and the teacher super-resolution reconstruction network respectively, a low-resolution image, each teacher feature map, a teacher super-resolution image, each student feature map, a student super-resolution image, a first low-frequency sub-band image, a first high-frequency sub-band image, a second low-frequency sub-band image and a second high-frequency sub-band image are obtained, and the loss information can be calculated.

[0090] S250: Iteratively optimize the student super-resolution reconstruction network according to the loss information until the preset conditions are met, and use the optimized student super-resolution reconstruction network as the super-resolution reconstruction model.

[0091] The student super-resolution reconstruction network is iteratively optimized according to the calculated loss information. Optionally, Adam can be used as a network optimizer. The learning rate of training is initially set to 1×10-4, and the number of iterations is set to 80. That is, after 80 iterative optimizations, the optimized student super-resolution reconstruction network is used as the super-resolution reconstruction model.

[0092] This method makes full use of the information in the image space and frequency domains and the relationship between them through wavelets and knowledge distillation, explores the multi-scale characteristics of the image after wavelet decomposition, obtains the deep potential features of the image, and realizes feature prior knowledge transfer learning, thereby reducing the number of network layers and network parameters of the super-resolution reconstruction model, thereby reducing the computational overhead of super-resolution reconstruction.

[0093] In a possible implementation, in order to extract richer image multi-scale features with fewer parameters, the student super-resolution reconstruction network may include a convolutional network and multiple multi-scale aggregation modules, see Figure 3 , the above step S230 may include the following steps:

[0094] S231: Input the first low-frequency sub-band image and the first high-frequency sub-band image into the convolutional network to obtain shallow features.

[0095] S232: Input the shallow features into each multi-scale aggregation module in turn to obtain the L-layer student feature map in the feature extraction process and the final deep features.

[0096] S233: Fusing the shallow features with the deep features to obtain fused wavelet coefficients, wherein the wavelet coefficients include the fused second low-frequency sub-band image and the second high-frequency sub-band image.

[0097] S234: Perform inverse wavelet reconstruction on the wavelet coefficients to obtain a student super-resolution image.

[0098] Each sub-band image is taken as the input of the student super-resolution reconstruction network and shares the same parameters. The student super-resolution reconstruction network first uses a convolutional network to extract the shallow features F of the wavelet domain image.s , and then learn the deep features F of the image through the multi-scale aggregation module d , by taking the shallow feature F s and deep features F d Fusion is performed to obtain the fused wavelet coefficient F p , and finally to F p Implement inverse wavelet reconstruction to obtain super-resolution image I SR The whole process can be expressed by the following formula:

[0099]

[0100] in, represents the upsampled image, f conv represents the convolution operation, f MAB Represents the multi-scale aggregation module operation, which is mainly used to extract the deep features of the image F d , W D represents wavelet decomposition, W R Represents inverse wavelet transform.

[0101] Optionally, f conv It is mainly composed of a convolutional network layer and a nonlinear activation layer, in which the nonlinear activation layer selects ReLU as the activation function. Specifically expressed as: Where W, b represent the weight and bias of the convolutional layer respectively, “*” represents the convolution operation, and the ReLU function is defined as max(0, x).

[0102] In a possible implementation, the output feature of the previous multi-scale aggregation module plus the input feature of the previous multi-scale aggregation module may be simultaneously input into the next multi-scale aggregation module, and the above step S232 may include the following steps:

[0103] S2321: Input the shallow features into the first multi-scale aggregation module to obtain the first deep features and multiple student feature maps.

[0104] S2322: sequentially inputting the input features of the previous multi-scale aggregation module and the first deep features into the next multi-scale aggregation module until the last multi-scale aggregation module to generate the final deep features.

[0105] By adding the input and output of a multi-scale aggregation module and inputting it into the next multi-scale aggregation module, richer multi-scale features of the image can be extracted.

[0106] Further, in order to expand the receptive field of the multi-scale aggregation module to extract features and capture more abundant image feature information, optionally, the multi-scale aggregation module can introduce channel splitting, channel aggregation and residual connection mechanisms based on residual learning. The multi-scale aggregation module may include a first fusion layer, a grouped convolution layer, N convolution layers, a channel shuffle layer and a second fusion layer. The above step S2321 may include the following: Figure 4 The following steps are shown:

[0107] S23211: Input the shallow features into the first fusion layer to obtain the first fusion features.

[0108] S23212: Input the first fusion feature into the group convolution layer to obtain N first group features.

[0109] S23213: Input the N first grouping features into N convolutional layers in sequence, and use the output of the previous convolutional layer as part of the input of the current convolutional layer to obtain N second grouping features with an increased receptive field.

[0110] S23214: Input the N second grouping features into the channel shuffling layer to obtain N third grouping features with rearranged channels.

[0111] S23215: Input the N third grouping features into the second fusion layer to obtain second fusion features.

[0112] S23216: Fuse the shallow feature with the second fusion feature to obtain the first deep feature.

[0113] S23217: Record the first fusion feature, N second grouping features, N third grouping features, the second fusion feature and the first deep feature as a student feature map.

[0114] Reference Figure 5 Optionally, the first fusion layer can be a 1×1 convolution layer, the number of N is 4, and the number of 4 convolution layers is 3×3. First, a 1×1 convolution layer is used to fuse the shallow features, and then the features are divided into 4 groups through a grouped convolution layer. For the features of each group, a 3×3 convolution layer is used to extract feature information. At the same time, in order to expand the receptive field of each group, the image features extracted from the current group are added to the next group. The process can be expressed by the following formula:

[0115]

[0116] Among them, x i ,y i , C i Represent the input features, output features, and convolutional layer of the th group respectively. In other words, the output of the previous group will be used as part of the input of the current group, so the current group will have a larger receptive field.

[0117] Afterwards, the extracted features are summarized and passed through a channel shuffle layer, i.e., a channel shuffle operation (ChannelShuffle) to overcome the side effects of channel splitting. Finally, the channel shuffled features are fused to obtain the first deep features.

[0118] In the feature introduction, the first fusion feature, N second grouping features, N third grouping features, the second fusion feature and the first deep feature are recorded as a student feature map for subsequent calculation of loss information.

[0119] The architecture of the teacher super-resolution reconstruction network is the same as that of the student super-resolution reconstruction network. The L-layer teacher feature map of the teacher super-resolution reconstruction network and the teacher super-resolution image acquisition process are consistent with those of the student super-resolution reconstruction network, and will not be repeated here.

[0120] The calculation of the loss information may take into account the supervision of the teacher super-resolution reconstruction network and data, which may help improve the performance of distillation. Optionally, step S240 may include the following steps:

[0121] S241: Based on the teacher supervision loss function, according to the teacher super-resolution image and the student super-resolution image, calculate the teacher supervision loss information of the teacher super-resolution reconstruction network supervising the student super-resolution reconstruction network;

[0122] S242: Based on the data supervision loss function, according to the student super-resolution image and the low-resolution image, the data supervision loss information of the low-resolution image on the student super-resolution reconstruction network is calculated.

[0123] S243: Calculate feature distillation loss information based on each teacher feature map and each student feature map.

[0124] S244: Based on the wavelet prediction loss function, calculate the wavelet prediction loss information of the student super-resolution image according to the first low-frequency sub-band image, the first high-frequency sub-band image, the second low-frequency sub-band image and the second high-frequency sub-band image.

[0125] S245: Based on the total loss function, the loss information is calculated according to the teacher supervision loss information, the data supervision loss information, the feature distillation loss information and the wavelet prediction loss information.

[0126] The teacher supervision loss function can be:

[0127]

[0128] in, represents the teacher super-resolution image, represents the student super-resolution image;

[0129] The data supervision loss function can be:

[0130]

[0131] in, represents the student super-resolution image, Represents a low-resolution image;

[0132] The wavelet prediction loss function can be:

[0133]

[0134] in, Respectively represent the first low-frequency sub-band image and the first high-frequency sub-band image, f represents the student super-resolution reconstruction network, Respectively represent the second low-frequency subband image and the second high-frequency subband image, and N represents the number of training samples in each batch;

[0135] The total loss function is:

[0136] L total =α1L TS +α2L DS +α3L kd +α4L wav

[0137] Among them, α1, α2, α3, α4 all represent loss weights, L kd represents the feature distillation loss function.

[0138] There are many ways to calculate the feature distillation loss information. In one possible implementation, the feature distillation loss information is calculated based on each teacher feature graph and each student feature graph, which may include the following steps:

[0139] S2431: Convert each teacher feature map and each student feature map into vectors and normalize them to obtain the corresponding teacher normalized column vector and student normalized column vector.

[0140] For each teacher feature map or student feature map, let F be a feature map whose dimension can be H×W (height×width), where each element F ij Represents the eigenvalue at position (i, j), converts the feature map F into a column vector, and reshapes F into a column vector f with a dimension of 1×(H×W). After scaling the elements of the column vector to a certain range, which can be [0, 1] or [-1, 1], the data has a uniform scale, and a normalized column vector is obtained:

[0141]

[0142] in, represents the square of the vector norm, and F is the teacher feature map or the student feature map.

[0143] S2432: Calculate the characteristic matrix of each teacher's normalized column vector and each student's normalized column vector to obtain the corresponding teacher characteristic matrix and student characteristic matrix respectively.

[0144] The calculation formula of the feature matrix is ​​as follows:

[0145]

[0146] in, express The transpose of represents the feature matrix M Line The elements corresponding to the columns, each element of the feature matrix M describes the spatial correlation between image pixels.

[0147] S2433: Based on the feature distillation loss function, the feature distillation loss information is calculated according to each teacher feature matrix and each student feature matrix.

[0148] The feature distillation loss function can be:

[0149]

[0150] Among them, |M| represents the number of elements of each teacher feature matrix and each student feature matrix, L represents the number of student feature maps or the number of teacher feature maps, represents the student feature matrix extracted from the student feature map of the lth layer, represents the student feature matrix extracted from the teacher feature map of the lth layer, and ||·||1 represents the l1 norm. In this embodiment, L can be set to 16 layers.

[0151] Furthermore, an embodiment of the present invention also provides an image super-resolution reconstruction method, Figure 6 A flowchart of an image super-resolution reconstruction method provided by an embodiment of the present invention is shown in FIG. Figure 6 , the execution subject of this method can be the above Figure 1 The electronic device 100 shown in FIG. 1 includes: Figure 6 The following steps are shown:

[0152] S310: Obtain a low-resolution image to be super-reconstructed.

[0153] S320: Input the low-resolution image into the super-resolution reconstruction model to obtain a super-resolution image after super-resolution reconstruction. The super-resolution reconstruction model is trained by the super-resolution reconstruction model training method described above.

[0154] The embodiment of the present invention uses the Chest Dataset chest X-ray data as the test object, compares the performance with the prior art method described in the background technology, and uses the Peak Signal Noise Ratio (PSNR) and Structural Similarity Index Measerment (SSIM) as evaluation indicators of image reconstruction performance. Table 1 and Table 2 respectively give the performance comparison results of the embodiment of the present invention and the prior art method in the ×3 super-resolution task and the ×4 super-resolution task. It can be seen from Table 1 and Table 2 that the embodiment of the present invention has obvious improvements in the two indicators of peak signal-to-noise ratio and structural similarity.

[0155] Table 1

[0156]

[0157]

[0158] Table 2

[0159]

[0160] In summary, the super-resolution reconstruction model training method, image super-resolution reconstruction method and electronic device provided by the embodiments of the present invention fully utilize the information in the image space domain and frequency domain and the relationship between them through wavelets and knowledge distillation, explore the multi-scale characteristics of the image after wavelet decomposition, obtain the deep potential features of the image, and realize feature prior knowledge transfer learning, thereby reducing the storage overhead of the network, improving the computing efficiency of the network, and improving the performance of the image super-resolution reconstruction model; through the feature extraction of the multi-scale aggregation module, richer image multi-scale features can be extracted with fewer parameters, increasing the receptive field of the network, and further improving the performance of image super-resolution reconstruction; through wavelet changes, wavelets can sparsely approximate the local features of the image, and wavelets are robust to noise. These wavelet characteristics can be used to capture the high-frequency detail information of the image and describe the context and texture information of the image at different levels.

[0161] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0163] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0164] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0165] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A super-resolution reconstruction model training method, characterized in that: The method comprises: Upsampling the low-resolution image and performing wavelet decomposition to obtain a first low-frequency sub-band image and a first high-frequency sub-band image; Inputting the first low-frequency sub-band image and the first high-frequency sub-band image into a trained teacher super-resolution reconstruction network to obtain an L-layer teacher feature map in the feature extraction process and a teacher super-resolution image after super-resolution reconstruction; Inputting the first low-frequency sub-band image and the first high-frequency sub-band image into a student super-resolution reconstruction network to obtain a processed second low-frequency sub-band image and a processed second high-frequency sub-band image, an L-layer student feature map in the feature extraction process, and a student super-resolution image after super-resolution reconstruction; wherein the architecture of the student super-resolution reconstruction network is the same as that of the teacher super-resolution reconstruction network, and the parameters of the student super-resolution reconstruction network are smaller than the parameters of the teacher super-resolution reconstruction network; Calculating loss information according to the low-resolution image, each of the teacher feature maps, the teacher super-resolution image, each of the student feature maps, the student super-resolution image, the first low-frequency sub-band image, the first high-frequency sub-band image, the second low-frequency sub-band image, and the second high-frequency sub-band image; The student super-resolution reconstruction network is iteratively optimized according to the loss information until a preset condition is met, and the optimized student super-resolution reconstruction network is used as a super-resolution reconstruction model.

2. The method according to claim 1, characterized in that The student super-resolution reconstruction network includes a convolutional network and a plurality of multi-scale aggregation modules. The first low-frequency sub-band image and the first high-frequency sub-band image are input into the student super-resolution reconstruction network to obtain a processed second low-frequency sub-band image and a second high-frequency sub-band image, an L-layer student feature map in the feature extraction process, and a student super-resolution image after super-resolution reconstruction, including: Inputting the first low-frequency sub-band image and the first high-frequency sub-band image into the convolutional network to obtain shallow features; Input the shallow features into each of the multi-scale aggregation modules in sequence to obtain the L-layer student feature map in the feature extraction process and the final deep features; Fusing the shallow features with the deep features to obtain fused wavelet coefficients; the wavelet coefficients include the fused second low-frequency sub-band image and the second high-frequency sub-band image; The wavelet coefficients are subjected to inverse wavelet reconstruction to obtain a student super-resolution image.

3. The method according to claim 2, characterized in that The shallow features are sequentially input into each of the multi-scale aggregation modules to obtain the L-layer student feature map in the feature extraction process and the final deep features, including: Inputting the shallow features into the first multi-scale aggregation module to obtain a first deep feature and a plurality of student feature maps; The input features of the previous multi-scale aggregation module and the first deep features are sequentially input into the next multi-scale aggregation module until the last multi-scale aggregation module to generate the final deep features.

4. The method according to claim 3, characterized in that The multi-scale aggregation module includes a first fusion layer, a grouped convolution layer, N convolution layers, a channel shuffle layer and a second fusion layer. The shallow features are input into the first multi-scale aggregation module to obtain the first deep features and multiple student feature maps, including: Inputting the shallow layer features into the first fusion layer to obtain a first fusion feature; Inputting the first fusion feature into the group convolution layer to obtain N first group features; Inputting N first grouping features into N convolutional layers in sequence, and using the output of the previous convolutional layer as part of the input of the current convolutional layer, to obtain N second grouping features with an increased receptive field; Inputting the N second grouping features into the channel shuffling layer to obtain N third grouping features with rearranged channels; Inputting the N third grouping features into the second fusion layer to obtain a second fusion feature; Fusing the shallow feature with the second fused feature to obtain a first deep feature; The first fusion feature, N of the second grouping features, N of the third grouping features, the second fusion feature and the first deep feature are recorded as a student feature map.

5. The method according to claim 1, characterized in that The loss information is calculated according to the low-resolution image, each of the teacher feature maps, the teacher super-resolution image, each of the student feature maps, the student super-resolution image, the first low-frequency sub-band image, the first high-frequency sub-band image, the second low-frequency sub-band image and the second high-frequency sub-band image, including: Based on the teacher supervision loss function, according to the teacher super-resolution image and the student super-resolution image, the teacher supervision loss information of the teacher super-resolution reconstruction network supervising the student super-resolution reconstruction network is calculated; Based on the data supervision loss function, according to the student super-resolution image and the low-resolution image, the data supervision loss information of the low-resolution image on the student super-resolution reconstruction network is calculated; Calculating feature distillation loss information according to each of the teacher feature graphs and each of the student feature graphs; Based on the wavelet prediction loss function, the wavelet prediction loss information of the student super-resolution image is calculated according to the first low-frequency sub-band image, the first high-frequency sub-band image, the second low-frequency sub-band image and the second high-frequency sub-band image; Based on the total loss function, the loss information is calculated according to the teacher supervision loss information, the data supervision loss information, the feature distillation loss information and the wavelet prediction loss information.

6. The method according to claim 5, characterized in that The feature distillation loss information is calculated according to each of the teacher feature graphs and each of the student feature graphs, including: Each of the teacher characteristic graphs and each of the student characteristic graphs are converted into vectors and normalized to obtain a corresponding teacher normalized column vector and a student normalized column vector; Calculate the characteristic matrix of each teacher normalized column vector and each student normalized column vector to obtain the corresponding teacher characteristic matrix and student characteristic matrix respectively; Based on the feature distillation loss function, the feature distillation loss information is calculated according to each of the teacher feature matrices and each of the student feature matrices.

7. The method according to claim 6, characterized in that The feature distillation loss function is: Wherein, |M| represents the number of elements of each teacher feature matrix and each student feature matrix, L represents the number of the student feature graphs or the number of the teacher feature graphs, represents the student feature matrix extracted from the student feature map of the lth layer, represents the student feature matrix extracted from the teacher feature map in the first layer, and ||·||1 represents Norm.

8. The method according to claim 5, characterized in that The teacher supervision loss function is: in, represents the teacher super-resolution image, represents the student super-resolution image; The data supervision loss function is: in, represents the student super-resolution image, representing the low-resolution image; The wavelet prediction loss function is: in, represent the first low-frequency sub-band image and the first high-frequency sub-band image respectively, f represents the student super-resolution reconstruction network, represent the second low-frequency sub-band image and the second high-frequency sub-band image respectively, and N represents the number of training samples in each batch; The total loss function is: L total =α1L TS +α2L DS +α3L kd +α4L wav Among them, α1, α2, α3, α4 all represent loss weights, L kd represents the feature distillation loss function.

9. An image super-resolution reconstruction method, characterized in that: The method comprises: Obtain a low-resolution image to be super-reconstructed; The low-resolution image is input into a super-resolution reconstruction model trained by the method according to any one of claims 1 to 8 to obtain a super-resolution image after super-resolution reconstruction.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 and / or claim 9 is implemented.

Images