Image super-resolution reconstruction method, system, device and storage medium
The multi-scale features of the image are extracted through sparse attention and shifted convolutional structures, and the problem of insufficient redundant information and performance in the prior art is solved, and a higher quality image super-resolution reconstruction is achieved.
Patent Information
- Application Number
- CN202310534807.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing deep learning-based image super-resolution methods generate redundant information when extracting multi-scale features of images, and cannot fully utilize image features, resulting in a degradation of super-resolution reconstruction performance.
Using sparse attention structure and shift convolution structure, multi-scale depth features of images are extracted through sparse processing and multi-scale window processing, and high-resolution network is trained through the minimum absolute deviation loss function, and iterative updates are performed to improve image super-resolution Rate reconstruction performance.
It effectively reduces redundant features, improves the reconstruction performance of image super-resolution, improves the reconstruction quality and computing efficiency of image details, and obtains higher quality super-resolution images.
Smart Images

Figure CN116797456B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to an image super-resolution reconstruction method, system, device and computer-readable storage medium. Background Art
[0002] Image super-resolution is a technique that uses mathematical algorithms to increase the resolution of low-resolution images. It can generate high-resolution images from original low-resolution images, thereby improving the quality and clarity of the image. This technique can be used in many fields such as medical imaging, video surveillance, aerospace, filmmaking, etc. In these fields, high-resolution images are essential to improve image quality and accuracy.
[0003] Traditional methods to improve image resolution are usually interpolation and reconstruction. Interpolation is to improve the resolution of an image by interpolating a low-resolution image. It is simple and fast, but it is less effective in improving image quality and details. Reconstruction is to improve the quality and resolution of an image by converting a low-resolution image into a high-resolution image. This method can be implemented using technologies such as deep learning and convolutional neural networks. It can effectively improve image quality and details, and has become the main research direction of image super-resolution.
[0004] In recent years, deep learning-based methods have been widely used in the field of image super-resolution. These methods mainly use convolutional neural networks to reconstruct high-resolution images. Common convolutional neural networks include SRCNN, VDSR, DRCN, ESPCN, EDSR, etc. Among them, SRCNN is the earliest method to use convolutional neural networks for image super-resolution. It maps low-resolution images to high-resolution space through a three-layer convolutional neural network. Subsequent algorithms have continued to develop on the basis of SRCNN, continuously improving the super-resolution quality of images. Due to the small number of network layers, the image reconstruction performance is also limited. To this end, Lim et al. proposed an EDSR model with a deeper network layer and a larger number of parameters. EDSR uses an enhanced residual structure to expand the depth of the network, which greatly improves the learning ability of the network and has made great progress in the image super-resolution task.
[0005] For a typical deep learning image super-resolution method, the features of low-resolution images have different degrees of effect on super-resolution reconstruction. Existing image super-resolution methods based on attention mechanism often generate a large amount of redundant information when extracting image features, and cannot fully utilize the multi-scale features of the image, thus degrading the image super-resolution reconstruction performance.
[0006] In summary, how to make full use of the multi-scale features of images, reduce redundant features, and improve the reconstruction performance of image super-resolution is a problem that needs to be solved. Summary of the invention
[0007] In order to solve the shortcomings of the existing methods, the present invention provides an image super-resolution reconstruction method, system, device and computer-readable storage medium, which can fully utilize the multi-scale features of the image and reduce redundant features, thereby improving the reconstruction performance of the image super-resolution.
[0008] The present invention provides an image super-resolution reconstruction method, which comprises the following steps:
[0009] S1, mapping the low-resolution image to be reconstructed into a high-dimensional space to extract shallow features of the image;
[0010] S2. Processing shallow features of the image using a sparse attention structure to extract single-scale deep features;
[0011] S3, setting three windows with different scale parameters, processing the single-scale depth features, and obtaining multi-scale depth features;
[0012] S4. Strengthen the multi-scale depth feature by using a shifted convolution structure to obtain a strengthened multi-scale depth feature;
[0013] S5, performing an upsampling operation on the enhanced multi-scale depth feature to obtain a super-resolution feature map;
[0014] S6. Training a high-resolution network composed of the sparse attention structure and the shifted convolution structure using a minimum absolute deviation loss function;
[0015] S7. Using the high-resolution network, iteratively update the super-resolution feature map and the high-resolution label image corresponding to the low-resolution image to be reconstructed to obtain a trained image super-resolution reconstruction model.
[0016] In one embodiment of the present invention, the step of processing the shallow features of the image using a sparse attention structure to extract single-scale deep features includes:
[0017] S21, performing window division on the shallow features of the image, dividing it into a plurality of windows of equal size;
[0018] S22, calculating the non-local attention of the shallow features of the image within the range of each window to obtain a similarity weight matrix;
[0019] S23, performing a sparse operation on the similarity weight matrix, filtering out irrelevant weight values in the similarity weight matrix, and using the remaining similarity weight matrix to perform weighted summation on each value in the image shallow feature matrix to obtain the single-scale deep feature.
[0020] In one embodiment of the present invention, extracting the single-scale depth feature includes:
[0021] The shallow feature matrix F0∈R C×H×W As input;
[0022] The shallow features of the image are divided into windows and divided into There are M×M windows of different sizes. At this time, the shallow feature matrix of the image is
[0023] Using the transfer matrix W implemented by 1×1 convolution q ,W k ,W v Performing linear transformation on the shallow features of the image respectively to obtain matrices Q, K and V;
[0024] The matrices Q and K are used to calculate the non-local attention within the range of each window to obtain the similarity weight matrix
[0025] The first K1 most relevant weights of the similarity weight matrix A are retained, and the irrelevant weights are assigned to negative infinity, and the sparse operation is completed to obtain the sparse weight matrix A. * ;
[0026] Filter the sparse weight matrix A using the Softmax activation function * , and weighted sum the filtered weight matrix pair matrix V to obtain the single-scale depth feature F;
[0027] The calculation formula is:
[0028] Q=W q F0,K=W k F0,V=W v F0
[0029] A=QK T
[0030] A * =Sparse(A)
[0031] F = Softmax(A * )V
[0032] Among them, K T represents the transpose of K, C, H, W represent the number of channels, length, and width of the shallow features of the image, respectively, L = M × M, and Sparse is the sparsification operation.
[0033] In one embodiment of the present invention, setting three windows with different scale parameters, processing the single-scale depth feature, and obtaining the multi-scale depth feature includes:
[0034] S31, setting three windows with different scale parameters;
[0035] S32, dividing the single-scale depth feature F into three parts f1, f2 and f3 in the channel dimension on average, and inputting them into the sparse attention structures of windows with different scale parameters respectively, to obtain three single-scale depth features F1, F2 and F3;
[0036] S33, the three single-scale depth features F1, F2 and F3 are fused in the channel dimension to obtain a multi-scale depth feature F 1 ;
[0037] The calculation formula is:
[0038] f1,f2,f3=Split(F)
[0039] F1, F2, F3 = SA (f1, f2, f3)
[0040] F 1 =Concat(F1,F2,F3)
[0041] Among them, Split is the channel segmentation function, SA is the sparse attention structure of windows with different scale parameters, and Concat is the channel concatenation function.
[0042] In one embodiment of the present invention, the step of strengthening the multi-scale depth feature by using a shifted convolution structure to obtain the enhanced multi-scale depth feature comprises:
[0043] S41, using the first 1×1 shift convolution to shift the multi-scale depth feature in the four directions of up, down, left, and right, and then performing a 1×1 convolution operation to obtain a shift feature matrix F s ;
[0044] S42, using 3×3 depth-separable convolution to process the shift feature matrix F s , and get the feature matrix F d ;
[0045] S43, using the second 1×1 shift convolution to process the feature matrix F d , and obtain the enhanced multi-scale deep feature F 2 ;
[0046] The calculation formula is:
[0047] F s =Shift(F 1 )
[0048] F d =Dw(Fs )
[0049] F 2 =Shift(F d )+F s
[0050] Among them, Shift represents 1×1 shift convolution, F 1 is a multi-scale deep feature, and Dw represents a 3×3 depth-separable convolution.
[0051] In one embodiment of the present invention, performing an upsampling operation on the enhanced multi-scale depth feature to obtain a super-resolution feature map includes:
[0052] After the enhanced multi-scale depth feature is convolved, the channel dimension is expanded to the square of the super-resolution magnification factor using a 1×1 convolution layer, and then the super-resolution feature map is obtained by reorganizing the pixels between multiple channels. The calculation formula is:
[0053] Y = PixelShuffle(Conv 1×1 (Conv 3×3 (F 2 )))
[0054] Among them, Y is the super-resolution feature map, PixelShuffle is the pixel reorganization function, Conv 1×1 is a 1×1 convolutional layer, Conv 3×3 is a 3×3 convolutional layer, F 2 To enhance multi-scale depth features.
[0055] In one embodiment of the present invention, the adopting of the minimum absolute deviation loss function to train the high-resolution network composed of the sparse attention structure and the shifted convolution structure comprises:
[0056] Calculating the error according to the minimum absolute deviation loss function, training the high-resolution network, and iterating repeatedly until the loss function converges;
[0057] The minimum absolute deviation loss function calculation formula is:
[0058]
[0059] Among them, N is the total number of training iterations, i is the number of this iteration, is a high-resolution label image, The generated super-resolution image.
[0060] The present invention also provides an image super-resolution reconstruction system, comprising:
[0061] A shallow feature extraction module is used to embed the low-resolution image to be reconstructed into a high-dimensional space and extract the shallow features of the image;
[0062] A sparse attention module, used to process the shallow features of the image using a sparse attention structure, extract and aggregate multi-scale features of the image, and obtain multi-scale deep features;
[0063] A shifted convolution module, used to enhance the multi-scale depth features by using a shifted convolution structure to obtain enhanced multi-scale depth features;
[0064] A super-resolution feature module, used for upsampling the enhanced multi-scale depth features to obtain a super-resolution feature map;
[0065] A training module is used to train a high-resolution network composed of the sparse attention structure and the shifted convolution structure using a minimum absolute deviation loss function; and using the high-resolution network, iteratively update the super-resolution feature map and the high-resolution label image corresponding to the low-resolution image to be reconstructed to obtain a trained image super-resolution reconstruction model.
[0066] The present invention also provides an image super-resolution reconstruction device, comprising a memory for storing a computer program; and a processor for implementing an image super-resolution reconstruction method as described in any one of the above when executing the computer program.
[0067] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the image super-resolution reconstruction method as described in any one of the above items is implemented.
[0068] The above technical solution of the present invention has the following advantages compared with the prior art:
[0069] The image super-resolution reconstruction method described in the present invention effectively extracts and aggregates multi-scale features of low-resolution images by designing a sparse attention structure. Multi-scale features help to reconstruct better image details, thereby improving the performance of image super-resolution; a large number of redundant features can be filtered out through the sparse attention structure block, thereby effectively reducing the computational complexity of the model. At the same time, a shifted convolution structure is introduced, and through continuous shifted convolution operations, the local detail features of the image can be fully extracted, thereby enhancing the representation ability of image features. Finally, the high-resolution network composed of the sparse attention structure and the shifted convolution structure can fully extract feature information of different scales of the image, thereby obtaining a higher quality super-resolution image. The image super-resolution reconstruction method described in the present invention can reconstruct low-resolution images in a real environment and obtain applied super-resolution images, which provides low-level technical support for further image tasks and contributes to technological innovation. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein
[0071] Figure 1 is a flow chart of an image super-resolution reconstruction method in a preferred embodiment of the present invention;
[0072] Figure 2 A specific flow chart of the image super-resolution reconstruction method in a preferred embodiment of the present invention;
[0073] Figure 3 A flowchart of a sparse attention module in a preferred embodiment of the present invention;
[0074] Figure 4 Flow chart of the shift convolution module in the preferred embodiment of the present invention;
[0075] Figure 5 4 is a structural block diagram of an image super-resolution reconstruction system in a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0076] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0077] It should be noted that image features are the semantic information of an image, including its contour, edge, texture, illumination, and color. After an image is subjected to operations such as convolution and attention weighting, an image feature matrix is obtained, which is the encoding of the image features. When image feature transformation operations are involved, such as convolution and attention weighting, we often use image feature matrices to express them. When expressing the nature of features, such as shallow features or deep features, we are accustomed to using image features to express them.
[0078] Reference Figure 1 As shown, the present invention provides an image super-resolution reconstruction method, comprising:
[0079] S1, mapping the low-resolution image to be reconstructed into a high-dimensional space to extract shallow features of the image;
[0080] S2. Processing shallow features of the image using a sparse attention structure to extract single-scale deep features;
[0081] S21, performing window division on the shallow features of the image, dividing it into a plurality of windows of equal size;
[0082] S22, calculating the non-local attention of the shallow features of the image within the range of each window to obtain a similarity weight matrix;
[0083] S23, performing a sparse operation on the similarity weight matrix, filtering irrelevant weight values in the similarity weight matrix, and using the remaining similarity weight matrix to perform a weighted summation on each value in the image shallow feature matrix to obtain the single-scale deep feature;
[0084] S3, setting three windows with different scale parameters, processing the single-scale depth features, and obtaining multi-scale depth features;
[0085] S31, setting three windows with different scale parameters;
[0086] S32, dividing the single-scale depth feature F into three parts f1, f2 and f3 in the channel dimension on average, and inputting them into the sparse attention structures of windows with different scale parameters respectively, to obtain three single-scale depth features F1, F2 and F3;
[0087] S33, the three single-scale depth features F1, F2 and F3 are fused in the channel dimension to obtain a multi-scale depth feature F 1 ;
[0088] S4. Strengthen the multi-scale depth feature by using a shifted convolution structure to obtain a strengthened multi-scale depth feature;
[0089] S41, using the first 1×1 shift convolution to shift the multi-scale depth feature in the four directions of up, down, left, and right, and then performing a 1×1 convolution operation to obtain a shift feature matrix F s ;
[0090] S42, using 3×3 depth-separable convolution to process the shift feature matrix F s , and get the feature matrix F d ;
[0091] S43, using the second 1×1 shift convolution to process the feature matrix F d , and obtain the enhanced multi-scale deep feature F 2 ;
[0092] S5, performing an upsampling operation on the enhanced multi-scale depth feature to obtain a super-resolution feature map;
[0093] After performing a convolution operation on the enhanced multi-scale depth feature, a 1×1 convolution layer is used to expand the channel dimension to the square of the super-resolution magnification factor, and then a super-resolution feature map is obtained by reorganizing pixels between multiple channels;
[0094] S6. Training a high-resolution network composed of the sparse attention structure and the shifted convolution structure using a minimum absolute deviation loss function;
[0095] Calculating the error according to the minimum absolute deviation loss function, training the high-resolution network, and iterating repeatedly until the loss function converges;
[0096] S7. Using the high-resolution network, iteratively update the super-resolution feature map and the high-resolution label image corresponding to the low-resolution image to be reconstructed to obtain a trained image super-resolution reconstruction model.
[0097] The image super-resolution reconstruction method described in the present invention effectively extracts and aggregates multi-scale features of low-resolution images by designing a sparse attention structure. Multi-scale features help to reconstruct better image details, thereby improving the performance of image super-resolution; a large number of redundant features can be filtered out through the sparse attention structure block, thereby effectively reducing the computational complexity of the model. At the same time, a shifted convolution structure is introduced, and through continuous shifted convolution operations, the local detail features of the image can be fully extracted, thereby enhancing the representation ability of image features. Finally, the high-resolution network composed of the sparse attention structure and the shifted convolution structure can fully extract feature information of different scales of the image, thereby obtaining a higher quality super-resolution image. The image super-resolution reconstruction method described in the present invention can reconstruct low-resolution images in a real environment and obtain applied super-resolution images, which provides low-level technical support for further image tasks and contributes to technological innovation.
[0098] In order to more clearly illustrate the technical solution of the present invention, the following is an explanation with specific data. Please refer to Figure 2-4 As shown in the figure, GT image, LR image, and SR image are the abbreviations of Ground Truth image, Low-Resolution image, and Super-Resolution image, respectively, and their Chinese names are high-resolution label image, low-resolution image, and super-resolution image, respectively.
[0099] The details are as follows:
[0100] 1. Extract shallow features of images:
[0101] As shown in formula (1), given the input low-resolution image X∈R 3×H×W First, a convolution layer with a convolution kernel size of 3×3, a step size of 1, a padding of 1, and a channel number of 60 is used to map the low-resolution image to a high-dimensional space with a channel dimension of 60, and extract the shallow feature F0 of the image;
[0102] F0=Conv 3×3 (X) (1)
[0103] Among them, Conv 3×3 It is a 3×3 convolution layer, 3×H×W means that the number of channels of the low-resolution image is 3, the length is H, and the width is W.
[0104] Second, use the sparse attention structure to process the shallow features of the image, extract and aggregate the multi-scale features of the image, and obtain the multi-scale deep features:
[0105] The shallow feature matrix F0∈R C×H×W As input;
[0106] The shallow features of the image are divided into windows and There are M×M windows of different sizes. At this time, the shallow feature matrix of the image is By calculating the non-local attention of shallow features of the image within each window, the computational complexity can be significantly reduced from quadratic time to linear time.
[0107] As shown in formula (2), the transfer matrix W implemented by 1×1 convolution is q ,W k ,W v Perform linear transformation on the shallow features F0 of the image respectively to obtain matrices Q, K and V;
[0108] As shown in formula (3), the matrices Q and K are used to calculate the non-local attention within the range of each window to obtain the similarity weight matrix
[0109] As shown in formula (4), the first K1 most relevant weights of the similarity weight matrix A are retained, and the irrelevant weights are assigned to negative infinity to complete the sparse operation, and the sparse weight matrix A is obtained. * ;
[0110] As shown in formula (5), the sparse weight matrix A is filtered using the Softmax activation function * , so that the irrelevant weights become sparsely distributed to 0, while retaining the weights of relevant information. The weighted sum of the filtered weight matrix V is performed to obtain the single-scale deep feature F;
[0111] Q=W q F0, K=W k F0, V=W v F0 (2)
[0112] A=QK T (3)
[0113] A * =Sparse(A) (4)
[0114] F = Softmax(A * )V (5)
[0115] Among them, K T represents the transpose of K, C, H, W represent the number of channels, length, and width of the shallow features of the image, respectively, L = M × M, and Sparse is the sparsification operation.
[0116] Since the obtained single-scale deep feature F only contains single-scale feature information, in order to extract multi-scale information of the image and enhance the representation ability of the feature, for this purpose:
[0117] Set three windows with different scale parameters, whose window sizes are 4×4, 8×8 and 16×16 respectively;
[0118] As shown in formula (6) and formula (7), the single-scale depth feature F is evenly divided into three parts f1, f2 and f3 in the channel dimension, and respectively input into the sparse attention structure with window sizes of 4×4, 8×8 and 16×16, and the depth features are further extracted to obtain three single-scale depth features F1, F2 and F3; the single-scale depth feature is evenly divided into three parts in the channel dimension, and the sparse attention is calculated in three single-scale windows in parallel, which can effectively reduce the computational complexity;
[0119] As shown in formula (8), the three single-scale depth features F1, F2 and F3 are fused in the channel dimension to obtain the multi-scale depth feature F 1 ;
[0120] f1,f2,f3=Split(F) (6)
[0121] F1, F2, F3 = SA (f1, f2, f3) (7)
[0122] F 1 =Concat(F1,F2,F3) (8)
[0123] Among them, Split is the channel segmentation function, SA is the sparse attention structure of windows with different scale parameters, and Concat is the channel concatenation function.
[0124] By processing low-resolution images with the above method, multi-scale features of low-resolution images can be effectively extracted and aggregated. Multi-scale features help reconstruct better image details, thereby improving the performance of image super-resolution; sparse attention blocks can filter out a large number of redundant features, thereby effectively reducing the computational complexity of the model.
[0125] 3. Use the shifted convolution structure to enhance the multi-scale depth features and obtain enhanced multi-scale depth features:
[0126] As shown in formula (9), the first 1×1 shifted convolution is used to transform the multi-scale depth feature F 1 Shift in the four directions of up, down, left, and right, and then perform a 1×1 convolution operation to obtain the shift feature matrix F s ;
[0127] As shown in formula (10), the shift feature matrix F is processed using a 3×3 depthwise separable convolution s , and get the feature matrix F d ; Through 3×3 depth-separable convolution, the local details of the features can be effectively supplemented;
[0128] As shown in formula (11), the feature matrix F is processed using a second 1×1 shift convolution d , and obtain the enhanced multi-scale deep feature F 2 ; Using the second 1×1 shift convolution can effectively expand the image receptive field and enhance the feature representation capability;
[0129] F s =Shift(F 1 ) (9)
[0130] F d =Dw(F s ) (10)
[0131] F 2 =Shift(F d )+F s (11)
[0132] Among them, Shift represents a 1×1 shifted convolution, and Dw represents a 3×3 depth-separable convolution.
[0133] By introducing the shifted convolution structure through the above method, through continuous shifted convolution operations, the local detail features of the image can be fully extracted, thereby enhancing the representation ability of image features.
[0134] Fourth, upsample the enhanced multi-scale depth features to obtain a super-resolution feature map:
[0135] As shown in formula (12), first, a 3×3 convolution kernel is used to enhance the multi-scale depth feature F after local detail enhancement. 2 Perform convolution to enhance the representation ability of high-frequency features and obtain high-frequency features;
[0136] Secondly, a 1×1 convolutional layer is used to expand the channel dimension of high-frequency features to the square of the super-resolution magnification factor;
[0137] Finally, the pixels between multiple channels are used to reorganize the high-frequency features whose channel dimension is expanded to the square of the super-resolution magnification factor, so as to obtain a super-resolution feature map;
[0138] Y = PixelShuffle(Conv 1×1 (Conv 3×3 (F 2 ))) (12)
[0139] Among them, Y is the super-resolution feature map, PixelShuffle is the pixel reorganization function, Conv 1×1 is a 1×1 convolutional layer, Conv 3×3 It is a 3×3 convolutional layer.
[0140] 5. Use the minimum absolute deviation loss function to train a high-resolution network consisting of a sparse attention structure and a shifted convolution structure:
[0141] BatchSize (Chinese meaning: the number of small batches of training sets) is set to 32, PatchSize (Chinese meaning: the size of the training set data after cropping) is set to 128, and the initial learning rate is set to 2×10 -4 , 2×10 per training 5 times, the learning rate decays by half;
[0142] As shown in formula (13), the error is calculated according to the minimum absolute deviation loss function, and the high-resolution network is trained, and iterated repeatedly until the loss function converges;
[0143]
[0144] Among them, N is the total number of training iterations, i is the number of this iteration, is a high-resolution label image, The generated super-resolution image.
[0145] 6. Using a high-resolution network, the super-resolution feature map is iteratively updated with the high-resolution label image corresponding to the low-resolution image to be reconstructed, thereby obtaining a trained image super-resolution reconstruction model.
[0146] Through the above technical scheme, the present invention proposes an image super-resolution reconstruction method based on a multi-scale sparse attention network based on deep learning technology. Compared with the existing super-resolution method, the method of the present invention can make full use of the multi-scale features of the image and reduce redundant features at the same time, thereby reconstructing a super-resolution image with more complete details and clearer visual effects, greatly improving the performance of image super-resolution.
[0147] Reference Figure 5As shown, the present invention also provides an image super-resolution reconstruction system, comprising:
[0148] A shallow feature extraction module 10 is used to embed the low-resolution image to be reconstructed into a high-dimensional space and extract shallow features of the image;
[0149] A sparse attention module 20, configured to process the shallow features of the image using a sparse attention structure, extract and aggregate multi-scale features of the image, and obtain multi-scale deep features;
[0150] A shift convolution module 30, used to enhance the multi-scale depth feature by using a shift convolution structure to obtain an enhanced multi-scale depth feature;
[0151] A super-resolution feature module 40 is used to perform an upsampling operation on the enhanced multi-scale depth feature to obtain a super-resolution feature map;
[0152] The training module 50 is used to train the high-resolution network composed of the sparse attention structure and the shifted convolution structure using the minimum absolute deviation loss function; and use the high-resolution network to iteratively update the super-resolution feature map and the high-resolution label image corresponding to the low-resolution image to be reconstructed to obtain a trained image super-resolution reconstruction model.
[0153] The system is used to implement the image super-resolution reconstruction method described in any one of the above embodiments, so the system also has all the advantages thereof and will not be described again.
[0154] The present invention also provides an image super-resolution reconstruction device, comprising a memory for storing a computer program; and a processor for implementing an image super-resolution reconstruction method as described in any one of the above embodiments when executing the computer program.
[0155] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the image super-resolution reconstruction method as described in any one of the above embodiments is implemented.
[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0160] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.
Claims
1. A method for super-resolution image reconstruction, characterized in that: include: S1, mapping the low-resolution image to be reconstructed into a high-dimensional space to extract shallow features of the image; S2. Processing shallow features of the image using a sparse attention structure to extract single-scale deep features; S3, setting three windows with different scale parameters, processing the single-scale depth features, and obtaining multi-scale depth features; S4. Strengthen the multi-scale depth feature by using a shifted convolution structure to obtain a strengthened multi-scale depth feature; S5, performing an upsampling operation on the enhanced multi-scale depth feature to obtain a super-resolution feature map; S6. Training a high-resolution network composed of the sparse attention structure and the shifted convolution structure using a minimum absolute deviation loss function; S7, using the high-resolution network, iteratively updating the super-resolution feature map and the high-resolution label image corresponding to the low-resolution image to be reconstructed, to obtain a trained image super-resolution reconstruction model; The step of processing the shallow features of the image using a sparse attention structure to extract single-scale deep features includes: S21, performing window division on the shallow features of the image, dividing it into a plurality of windows of equal size; S22, calculating the non-local attention of the shallow features of the image within the range of each window to obtain a similarity weight matrix; S23, performing a sparse operation on the similarity weight matrix, filtering irrelevant weight values in the similarity weight matrix, and using the remaining similarity weight matrix to perform a weighted summation on each value in the image shallow feature matrix to obtain the single-scale deep feature; The method of using a shifted convolution structure to strengthen the multi-scale depth feature to obtain the enhanced multi-scale depth feature includes: S41, using the first 1×1 shift convolution to shift the multi-scale depth feature in the four directions of up, down, left, and right, and then performing a 1×1 convolution operation to obtain a shift feature matrix F s ; S42, using 3×3 depth-separable convolution to process the shift feature matrix F s , and get the feature matrix F d ; S43, using the second 1×1 shift convolution to process the feature matrix F d , and obtain the enhanced multi-scale deep feature F 2 ; The calculation formula is: F s =Shift(F 1 ) F d =Dw(F s ) F 2 =Shift(F d )+F s Among them, Shift represents 1×1 shift convolution, F 1 is a multi-scale deep feature, and Dw represents a 3×3 depth-separable convolution.
2. The image super-resolution reconstruction method according to claim 1, characterized in that: Extracting the single-scale depth feature includes: The shallow feature matrix F0∈R C×H×W As input; The shallow features of the image are divided into windows and divided into There are M×M windows of different sizes. At this time, the shallow feature matrix of the image is Using the transfer matrix W implemented by 1×1 convolution q ,W k ,W v Performing linear transformation on the shallow features of the image respectively to obtain matrices Q, K and V; The matrices Q and K are used to calculate the non-local attention within the range of each window to obtain the similarity weight matrix The first K1 most relevant weights of the similarity weight matrix A are retained, and the irrelevant weights are assigned to negative infinity, and the sparse operation is completed to obtain the sparse weight matrix A. * ; Filter the sparse weight matrix A using the Softmax activation function * , and use the filtered weight matrix to perform weighted summation on the matrix V to obtain the single-scale depth feature F; The calculation formula is: Q=W q F0,K=W k F0,V=W v F0 A=QK T A * =Sparse(A) F=Softmax(A * )V Among them, K T represents the transpose of K, C, H, W represent the number of channels, length, and width of the shallow features of the image, respectively, L = M × M, and Sparse is the sparsification operation.
3. The image super-resolution reconstruction method according to claim 1, characterized in that: The step of setting three windows with different scale parameters, processing the single-scale depth features, and obtaining the multi-scale depth features comprises: S31, setting three windows with different scale parameters; S32, dividing the single-scale depth feature F into three parts f1, f2 and f3 in the channel dimension on average, and inputting them into the sparse attention structures of windows with different scale parameters respectively, to obtain three single-scale depth features F1, F2 and F3; S33, the three single-scale depth features F1, F2 and F3 are fused in the channel dimension to obtain a multi-scale depth feature F 1 ; The calculation formula is: f1,f2,f3=Split(F) F1, F2, F3 = SA (f1, f2, f3) F 1 =Concat(F1,F2,F3) Among them, Split is the channel segmentation function, SA is the sparse attention structure of windows with different scale parameters, and Concat is the channel concatenation function.
4. The image super-resolution reconstruction method according to claim 1, characterized in that: The upsampling operation of the enhanced multi-scale depth feature to obtain a super-resolution feature map comprises: After the enhanced multi-scale depth feature is convolved, the channel dimension is expanded to the square of the super-resolution magnification factor using a 1×1 convolution layer, and then the super-resolution feature map is obtained by reorganizing the pixels between multiple channels. The calculation formula is: Y=PixelShuffle(Conv 1×1 (Conv 3×3 (F 2 ))) Among them, Y is the super-resolution feature map, PixelShuffle is the pixel reorganization function, Conv 1×1 is a 1×1 convolutional layer, Conv 3×3 is a 3×3 convolutional layer, F 2 To enhance multi-scale depth features.
5. The image super-resolution reconstruction method according to claim 1, characterized in that: The adopting of the minimum absolute deviation loss function to train the high-resolution network composed of the sparse attention structure and the shifted convolution structure includes: Calculating the error according to the minimum absolute deviation loss function, training the high-resolution network, and iterating repeatedly until the loss function converges; The minimum absolute deviation loss function calculation formula is: Among them, N is the total number of training iterations, i is the number of this iteration, is a high-resolution label image, The generated super-resolution image.
6. An image super-resolution reconstruction system, characterized in that: include: A shallow feature extraction module is used to embed the low-resolution image to be reconstructed into a high-dimensional space and extract the shallow features of the image; A sparse attention module, used for processing shallow features of the image using a sparse attention structure to extract single-scale deep features; Three windows with different scale parameters are set to process the single-scale deep features to obtain multi-scale deep features; wherein, the shallow features of the image are processed using a sparse attention structure, and the single-scale deep features are extracted, including: performing window division on the shallow features of the image, dividing them into a plurality of windows of equal size; calculating the non-local attention of the shallow features of the image within the range of each of the windows to obtain a similarity weight matrix; performing a sparse operation on the similarity weight matrix, filtering irrelevant weight values in the similarity weight matrix, and using the remaining similarity weight matrix to perform weighted summation on each value in the shallow feature matrix of the image to obtain the single-scale deep features; A shift convolution module is used to strengthen the multi-scale depth feature by using a shift convolution structure to obtain a strengthened multi-scale depth feature; wherein, strengthening the multi-scale depth feature by using a shift convolution structure to obtain a strengthened multi-scale depth feature comprises: using a first 1×1 shift convolution to shift the multi-scale depth feature in four directions of up, down, left and right, and then performing a 1×1 convolution operation to obtain a shift feature matrix F s ; Use 3×3 depth-separable convolution to process the shifted feature matrix F s , and get the feature matrix F d ; Use the second 1×1 shift convolution to process the feature matrix F d , and obtain the enhanced multi-scale deep feature F 2 ; The calculation formula is: F s =Shift(F 1 ) F d =Dw(F s ) F 2 =Shift(F d )+F s Among them, Shift represents 1×1 shift convolution, F 1 It is a multi-scale deep feature, Dw represents a 3×3 depth-separable convolution; A super-resolution feature module, used for upsampling the enhanced multi-scale depth features to obtain a super-resolution feature map; A training module is used to train a high-resolution network composed of the sparse attention structure and the shifted convolution structure using a minimum absolute deviation loss function; and using the high-resolution network, iteratively update the super-resolution feature map and the high-resolution label image corresponding to the low-resolution image to be reconstructed to obtain a trained image super-resolution reconstruction model.
7. An image super-resolution reconstruction device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement an image super-resolution reconstruction method as described in any one of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, an image super-resolution reconstruction method as described in any one of claims 1 to 5 is implemented.
