Image super-resolution reconstruction method and related device based on image reconstruction model

Abstract

The embodiments of the present application disclose a method for image super-resolution reconstruction based on an image reconstruction model and a related device, wherein the method is based on the image reconstruction model, and the image reconstruction model includes a first reconstruction network, a supervised attention (SAM) model, and a second reconstruction network; the method includes: processing a low-resolution (LR) image through the first reconstruction network to generate a first high-resolution (HR) image; processing the first HR image through the SAM model to generate second feature data; and processing the first HR image and the second feature data through the second reconstruction network to output a second HR image. The embodiments of the present application obtain a fine image super-resolution reconstruction result by performing a secondary reconstruction on the LR image and outputting a second HR image with enhanced boundary information and texture details.

Description

Technical Field

[0001] The present application relates to computer technology and is applied to the field of image processing, and in particular to an image super-resolution reconstruction method and related devices based on an image reconstruction model. Background Art

[0002] Image super-resolution reconstruction technology is a processing technique that converts low-resolution (LR) images into high-resolution (HR) images. It is widely used in fields such as medicine, remote sensing, and video surveillance. Currently, deep learning-based image super-resolution reconstruction techniques typically extract features from LR images and then upsample them to produce HR images. Due to the rapid conversion from low-resolution to high-resolution, image boundary information and texture details are easily lost, making it impossible to achieve fine super-resolution reconstruction results.

[0003] Therefore, how to obtain fine image super-resolution reconstruction results is a technical problem being studied by those skilled in the art. Summary of the Invention

[0004] The embodiments of the present application disclose an image super-resolution reconstruction method and related devices based on an image reconstruction model, which can enhance the boundary information and texture details of an image and obtain fine image super-resolution reconstruction results.

[0005] In a first aspect, an embodiment of the present application provides an image super-resolution reconstruction method based on an image reconstruction model, wherein the image reconstruction model includes a first reconstruction network, a supervised attention module (SAM) model, and a second reconstruction network, and the method includes:

[0006] Performing feature extraction on the low-resolution LR image through the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data, wherein each pixel in the LR image has a grayscale value corresponding to three channels (red, green, and blue, RGB);

[0007] Performing feature restoration on the first feature data through the upsampling layer and the second convolutional layer in the first reconstruction network to generate a first high-resolution HR image;

[0008] Performing feature calibration on the first high-resolution image using the SAM model to generate second feature data;

[0009] performing feature extraction on the first HR image through a third convolutional layer in the second reconstruction network to generate third feature data;

[0010] concatenating the third feature data with the second feature data to generate fourth feature data;

[0011] performing feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data;

[0012] performing feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data;

[0013] splicing the sixth feature data with the second feature data to generate seventh feature data;

[0014] The sixth convolutional layer in the second reconstruction network is used to perform feature restoration on the seventh feature data, and a second HR image is output.

[0015] In the above method, due to the rapid conversion from low resolution to high resolution, the first HR image generated by the first reconstruction network of the LR image weakens the original boundary information and texture details in the LR image. In order to solve this problem, the above method calculates the attention score of the first HR image through the SAM model, which is used to perform feature calibration on the first HR image to generate second feature data, and then the first HR image is spliced ​​twice with the second feature data through the second reconstruction network. This not only retains the features of the LR image extracted by the first reconstruction network, but also enables the second reconstruction network to focus on the features in the LR image that are more important to the reconstruction effect, thereby outputting a second HR image with enhanced boundary information and texture details. In other words, the above method performs secondary reconstruction on the LR image through the image reconstruction model to obtain a fine image super-resolution reconstruction result.

[0016] In combination with the first aspect, in one possible implementation, extracting features from the LR image using the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data includes:

[0017] Performing feature extraction on the LR image through the first convolutional layer to generate eighth feature data;

[0018] Inputting the eighth feature data into the first residual network, wherein the first residual network includes N residual blocks, and the N residual blocks are used to perform iterative feature extraction on the input eighth feature data, where N is an integer greater than 1;

[0019] The output data of the first residual network is fused with the LR image to generate the first feature data.

[0020] In the above method, the first convolutional layer is used to perform preliminary feature extraction on the LR image, and the first residual network is used to perform deep feature extraction on the LR image. Therefore, the first feature information generated by the LR image after processing by the first convolutional layer and the first residual network fuses the information of the LR image and the rich feature information in the LR image, which is conducive to feature restoration by the subsequent upsampling layer and the convolution layer to obtain a feature-enhanced first HR image.

[0021] In combination with the first aspect, or any of the foregoing possible implementations of the first aspect, in another possible implementation, the SAM model includes a seventh convolutional layer, an eighth convolutional layer, and an S-shaped growth curve Sigmoid activation layer; and performing feature calibration on the first HR image using the SAM model to generate second feature data includes:

[0022] performing feature extraction on the first HR image through the seventh convolutional layer to generate ninth feature data;

[0023] splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first HR image to generate eleventh feature data;

[0024] Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data;

[0025] multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data;

[0026] The thirteenth feature data is fused with the first HR image to generate the second feature data.

[0027] In the above method, the twelfth feature data includes attention scores for multiple pixel-level features, representing their importance to the reconstruction result. Therefore, by performing a weighted calculation of the attention scores on the first HR image using the SAM model, the above method can enhance important features in the first HR image and generate second feature data, which is then used to supervise the second reconstruction model to focus on the important features and output a second HR image with enhanced boundary information and texture details.

[0028] In combination with the first aspect, or any of the foregoing possible implementations of the first aspect, in another possible implementation, the extracting features of the fourth feature data using the fourth convolutional layer and the second residual network in the second reconstruction network to generate the fifth feature data includes:

[0029] performing feature extraction on the fourth feature data through the fourth convolutional layer to generate fourteenth feature data;

[0030] Inputting the fourteenth feature data into the second residual network, wherein the second residual network includes M connected residual blocks, and the M residual blocks are used to perform iterative feature extraction on the input fourteenth feature data, where M is an integer greater than 1;

[0031] The output data of the second residual network is fused with the fourteenth feature data to generate the fifth feature data.

[0032] In the above method, the fourth feature data is obtained by splicing the features of the first HR image and the enhanced features obtained by processing the first HR image through the SAM model. The fourth convolutional layer and the second residual network are used to further fuse the feature information in the fourth feature data to facilitate subsequent processing.

[0033] In a second aspect, an embodiment of the present application provides a training method for an image reconstruction model, wherein the image reconstruction model includes a first reconstruction network, a supervised attention SAM model, and a second reconstruction network, and the method includes:

[0034] Performing feature extraction on the low-resolution (LR) sample image using the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data, wherein each pixel in the LR sample image has grayscale values ​​corresponding to three channels (red, green, and blue, RGB), and the LR sample image is an image obtained by degrading the target high-resolution (HR) sample image;

[0035] Perform feature restoration on the first feature data through the upsampling layer and the second convolution layer in the first reconstruction network to generate a first HR sample image;

[0036] Performing feature calibration on the first HR sample image using the SAM model to generate second feature data;

[0037] performing feature extraction on the first HR sample image through the third convolutional layer in the second reconstruction network to generate third feature data;

[0038] concatenating the third feature data with the second feature data to generate fourth feature data;

[0039] performing feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data;

[0040] performing feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data;

[0041] splicing the sixth feature data with the second feature data to generate seventh feature data;

[0042] Performing feature restoration on the seventh feature data through the sixth convolutional layer in the second reconstruction network, and outputting a second HR sample image;

[0043] Parameters in the first reconstruction network, the SAM model, and the second reconstruction network are optimized according to the first HR sample image, the second HR sample image, and the target HR sample image.

[0044] The above method continuously optimizes the parameters of the image reconstruction model by comparing the differences among the first HR sample image, the second HR sample image and the target HR sample image, thereby improving the accuracy of LR image reconstruction.

[0045] Optionally, optimizing parameters in the first reconstruction network, the SAM model, and the second reconstruction network according to the first HR sample image, the second HR sample image, and the target HR sample image includes:

[0046] Calculating the difference between the first HR sample image, the second HR sample image and the target HR sample image by using a loss function;

[0047] Optimizing parameters in the first reconstruction network, the SAM model, and the second reconstruction network until the gap value is smaller than a preset threshold.

[0048] In combination with the second aspect, in a possible implementation, the first residual network includes N connected residual blocks, wherein the first residual block includes two convolutional layers, an instance normalization (IN) layer, and two rectified linear unit (ReLU) activation layers, the input data of the IN layer includes half of the feature data of the output data of the first convolutional layer, and the input data of the first ReLU activation layer includes the concatenation of the other half of the feature data of the output data of the first convolutional layer and the output data of the IN layer; the output data of the first residual block includes the fusion data of the output data of the second ReLU activation layer in the first residual block and the input data of the first residual block, which is used as the input data of the next residual block; the first residual block is any one of the N residual blocks, where N is an integer greater than 1;

[0049] The second residual network includes M connected residual blocks, wherein the second residual block includes two convolutional layers, an instance normalization (IN) layer and two rectified linear unit (ReLU) activation layers, the input data of the IN layer includes half of the feature data of the output data of the first convolutional layer, and the input data of the first ReLU activation layer includes the concatenation of the other half of the feature data of the output data of the first convolutional layer and the output data of the IN layer; the output data of the second residual block includes the fusion data of the output data of the second ReLU activation layer in the second residual block and the input data of the second residual block, which is used as the input data of the next residual block; the second residual block is any one of the M residual blocks, and M is an integer greater than 1.

[0050] In the above method, the input data of each residual block in the first residual network and the second residual network is the output data of the previous unit. When the parameters of the previous unit change, the distribution of the input data will change significantly, which will easily lose the feature information in the original input data and reduce the convergence speed during model training. To solve this problem, each residual block in the first residual network and the second residual network of the above method is provided with an IN layer, which is used to normalize half of the feature data of the input data obtained by the first convolutional layer. That is to say, each residual block in the first residual network and the second residual network performs half-instance normalization on the input data. This can not only retain the feature information in the original input data and improve the super-resolution reconstruction effect of the image reconstruction model, but also accelerate the convergence speed of the image reconstruction model and improve the training speed of the image reconstruction model compared to batch normalization or full instance normalization.

[0051] In combination with the second aspect, or any of the foregoing possible implementations of the second aspect, in another possible implementation, extracting features from the LR sample image using the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data includes:

[0052] Performing feature extraction on the LR sample image through the first convolutional layer to generate eighth feature data;

[0053] Inputting the eighth feature data into the first residual network, wherein the first residual network includes N connected residual blocks, and the N residual blocks are used to perform iterative feature extraction on the input eighth feature data, where N is an integer greater than 1;

[0054] The output data of the first residual network is fused with the LR sample image to generate the first feature data.

[0055] In combination with the second aspect, or any of the foregoing possible implementations of the second aspect, in another possible implementation, the SAM model includes a seventh convolutional layer, an eighth convolutional layer, and an S-shaped growth curve Sigmoid activation layer; and performing feature calibration on the first HR sample image using the SAM model to generate the second feature data includes:

[0056] performing feature extraction on the first HR sample image through the seventh convolutional layer to generate ninth feature data;

[0057] splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first HR sample image to generate eleventh feature data;

[0058] Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data;

[0059] multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data;

[0060] The thirteenth feature data is fused with the first HR sample image to generate the second feature data.

[0061] In combination with the second aspect, or any of the foregoing possible implementations of the second aspect, in another possible implementation, the extracting features of the fourth feature data using the fourth convolutional layer and the second residual network in the second reconstruction network to generate the fifth feature data includes:

[0062] performing feature extraction on the fourth feature data through the fourth convolutional layer to generate fourteenth feature data;

[0063] Inputting the fourteenth feature data into the second residual network, wherein the second residual network includes M connected residual blocks, and the M residual blocks are used to perform iterative feature extraction on the input fourteenth feature data, where M is an integer greater than 1;

[0064] The output data of the second residual network is fused with the fourteenth feature data to generate the fifth feature data.

[0065] In a third aspect, an embodiment of the present application provides an image super-resolution reconstruction device, the device comprising:

[0066] A first processing unit is configured to perform feature extraction on the low-resolution LR image through a first convolutional layer and a first residual network in a first reconstruction network to generate first feature data, wherein each pixel in the LR image has a grayscale value corresponding to three channels (red, green, and blue (RGB)).

[0067] a second processing unit, configured to perform feature restoration on the first feature data through an upsampling layer and a second convolutional layer in the first reconstruction network to generate a first high-resolution HR image;

[0068] a third processing unit, configured to perform feature calibration on the first high-resolution image using a SAM model to generate second feature data;

[0069] a fourth processing unit, configured to perform feature extraction on the first HR image through a third convolutional layer in the second reconstruction network to generate third feature data;

[0070] a first splicing unit, configured to splice the third feature data with the second feature data to generate fourth feature data;

[0071] a fifth processing unit, configured to perform feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data;

[0072] a sixth processing unit, configured to perform feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data;

[0073] a second splicing unit, configured to splice the sixth feature data with the second feature data to generate seventh feature data;

[0074] An output unit is used to perform feature restoration on the seventh feature data through the sixth convolutional layer in the second reconstruction network, and output a second HR image.

[0075] In conjunction with the third aspect, in one possible implementation, in the aspect of extracting features from the LR image through the first convolutional layer and the first residual network in the first reconstruction network to generate the first feature data, the first processing unit is specifically configured to:

[0076] Performing feature extraction on the LR image through the first convolutional layer to generate eighth feature data;

[0077] Inputting the eighth feature data into the first residual network, wherein the first residual network includes N connected residual blocks, and the N residual blocks are used to perform iterative feature extraction on the input eighth feature data, where N is an integer greater than 1;

[0078] The output data of the first residual network is fused with the LR image to generate the first feature data.

[0079] In combination with the third aspect, or any of the foregoing possible implementations of the third aspect, in another possible implementation, the SAM model includes a seventh convolutional layer, an eighth convolutional layer, and an S-shaped growth curve Sigmoid activation layer; in performing feature calibration on the first HR image using the SAM model to generate the second feature data, the third processing unit is specifically configured to:

[0080] performing feature extraction on the first HR image through the seventh convolutional layer to generate ninth feature data;

[0081] splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first HR image to generate eleventh feature data;

[0082] Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data;

[0083] multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data;

[0084] The thirteenth feature data is fused with the first HR image to generate the second feature data.

[0085] In combination with the third aspect, or any of the foregoing possible implementations of the third aspect, in another possible implementation, in the aspect of extracting features from the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data, the fifth processing unit is specifically configured to:

[0086] performing feature extraction on the fourth feature data through the fourth convolutional layer to generate fourteenth feature data;

[0087] Inputting the fourteenth feature data into the second residual network, wherein the second residual network includes M connected residual blocks, and the M residual blocks are used to perform iterative feature extraction on the input fourteenth feature data, where M is an integer greater than 1;

[0088] The output data of the second residual network is fused with the fourteenth feature data to generate the fifth feature data.

[0089] In a fourth aspect, an embodiment of the present application provides a training device for an image reconstruction model, wherein the image reconstruction model includes a first reconstruction network, a supervised attention SAM model, and a second reconstruction network, and the device includes:

[0090] a first processing unit, configured to perform feature extraction on a low-resolution (LR) sample image using a first convolutional layer and a first residual network in the first reconstruction network to generate first feature data, wherein each pixel in the LR sample image has a grayscale value corresponding to three channels (red, green, and blue, RGB), and the LR sample image is an image obtained by degrading a target high-resolution (HR) sample image;

[0091] a second processing unit, configured to perform feature restoration on the first feature data through an upsampling layer and a second convolutional layer in the first reconstruction network to generate a first HR sample image;

[0092] a third processing unit, configured to perform feature calibration on the first HR sample image using the SAM model to generate second feature data;

[0093] a fourth processing unit, configured to perform feature extraction on the first HR sample image through the third convolutional layer in the second reconstruction network to generate third feature data;

[0094] a first splicing unit, configured to splice the third feature data with the second feature data to generate fourth feature data;

[0095] a fifth processing unit, configured to perform feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data;

[0096] a sixth processing unit, configured to perform feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data;

[0097] a second splicing unit, configured to splice the sixth feature data with the second feature data to generate seventh feature data;

[0098] an output unit, configured to perform feature restoration on the seventh feature data through the sixth convolutional layer in the second reconstruction network, and output a second HR sample image;

[0099] An optimization unit is used to optimize parameters in the first reconstruction network, the SAM model and the second reconstruction network according to the first HR sample image, the second HR sample image and the target HR sample image.

[0100] Optionally, in optimizing the parameters of the first reconstruction network, the SAM model, and the second reconstruction network according to the first HR sample image, the second HR sample image, and the target HR sample image, the optimization unit is specifically configured to:

[0101] Calculating the difference between the first HR sample image, the second HR sample image and the target HR sample image by using a loss function;

[0102] Optimizing parameters in the first reconstruction network, the SAM model, and the second reconstruction network until the gap value is smaller than a preset threshold.

[0103] In combination with the fourth aspect, in a possible implementation, the first residual network includes N connected residual blocks, wherein the first residual block includes two convolutional layers, an instance normalization (IN) layer, and two rectified linear unit (ReLU) activation layers, the input data of the IN layer includes half of the feature data of the output data of the first convolutional layer, and the input data of the first ReLU activation layer includes the concatenation of the other half of the feature data of the output data of the first convolutional layer and the output data of the IN layer; the output data of the first residual block includes the fusion data of the output data of the second ReLU activation layer in the first residual block and the input data of the first residual block, which is used as the input data of the next residual block; the first residual block is any one of the N residual blocks, where N is an integer greater than 1;

[0104] The second residual network includes M connected residual blocks, wherein the second residual block includes two convolutional layers, an instance normalization (IN) layer and two rectified linear unit (ReLU) activation layers, the input data of the IN layer includes half of the feature data of the output data of the first convolutional layer, and the input data of the first ReLU activation layer includes the concatenation of the other half of the feature data of the output data of the first convolutional layer and the output data of the IN layer; the output data of the second residual block includes the fusion data of the output data of the second ReLU activation layer in the second residual block and the input data of the second residual block, which is used as the input data of the next residual block; the second residual block is any one of the M residual blocks, and M is an integer greater than 1.

[0105] In combination with the fourth aspect, or any of the foregoing possible implementations of the fourth aspect, in another possible implementation, in the aspect of extracting features from the LR sample image through the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data, the first processing unit is specifically configured to:

[0106] Performing feature extraction on the LR sample image through the first convolutional layer to generate eighth feature data;

[0107] Inputting the eighth feature data into the first residual network, wherein the first residual network includes N connected residual blocks, and the N residual blocks are used to perform iterative feature extraction on the input eighth feature data, where N is an integer greater than 1;

[0108] The output data of the first residual network is fused with the LR sample image to generate the first feature data.

[0109] In combination with the fourth aspect, or any of the foregoing possible implementations of the fourth aspect, in another possible implementation, the SAM model includes a seventh convolutional layer, an eighth convolutional layer, and an S-shaped growth curve Sigmoid activation layer; in performing feature calibration on the first HR sample image using the SAM model to generate the second feature data, the third processing unit is specifically configured to:

[0110] performing feature extraction on the first HR sample image through the seventh convolutional layer to generate ninth feature data;

[0111] splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first HR sample image to generate eleventh feature data;

[0112] Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data;

[0113] multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data;

[0114] The thirteenth feature data is fused with the first HR sample image to generate the second feature data.

[0115] In combination with the fourth aspect, or any of the foregoing possible implementations of the fourth aspect, in another possible implementation, in the aspect of extracting features from the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data, the fifth processing unit is specifically configured to:

[0116] performing feature extraction on the fourth feature data through the fourth convolutional layer to generate fourteenth feature data;

[0117] Inputting the fourteenth feature data into the second residual network, wherein the second residual network includes M connected residual blocks, and the M residual blocks are used to perform iterative feature extraction on the input fourteenth feature data, where M is an integer greater than 1;

[0118] The output data of the second residual network is fused with the fourteenth feature data to generate the fifth feature data.

[0119] In a fifth aspect, an embodiment of the present application provides an image super-resolution reconstruction device, which includes a processor, a memory, and a communication interface, wherein the communication interface is used to perform receiving and / or sending operations under the control of the processor, the memory is used to store a computer program, and the processor is used to call the computer program to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0120] In the sixth aspect, an embodiment of the present application provides a training device for an image reconstruction model, which includes a processor, a memory, and a communication interface, wherein the communication interface is used to perform receiving and / or sending operations under the control of the processor, the memory is used to store a computer program, and the processor is used to call the computer program to implement the method described in the second aspect or any possible implementation method of the second aspect.

[0121] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when running on a processor, is used to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0122] In an eighth aspect, an embodiment of the present application provides another computer-readable storage medium, which stores a computer program, which, when executed on a processor, is used to implement the method described in the second aspect or any possible implementation of the second aspect.

[0123] The beneficial effects of the technical methods provided in the second to eighth aspects of this application can refer to the beneficial effects of the technical solution in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] The following is a brief introduction to the drawings required for describing the embodiments of this application.

[0125] Figure 1 Schematic diagram of the architecture of an image super-resolution reconstruction system provided in an embodiment of the present application;

[0126] Figure 2 1 is a flow chart of an image super-resolution reconstruction method provided in an embodiment of the present application;

[0127] Figure 3 This is a schematic diagram of a process for reconstructing a network provided in an embodiment of the present application;

[0128] Figure 4This is a schematic diagram of an algorithm for a convolutional layer provided in an embodiment of the present application;

[0129] Figure 5 This is a schematic diagram of an algorithm for a residual block provided in an embodiment of the present application;

[0130] Figure 6 This is a flow chart of a supervised attention model provided in an embodiment of the present application;

[0131] Figure 7 This is another flowchart of a network reconstruction process provided by an embodiment of the present application;

[0132] Figure 8 1 is a flow chart of a method for training an image reconstruction model provided in an embodiment of the present application;

[0133] Figure 9 This is a schematic diagram of another residual block algorithm provided in an embodiment of the present application;

[0134] Figure 10 1 is a structural diagram of an image super-resolution reconstruction device 10 provided in an embodiment of the present application;

[0135] Figure 11 1 is a structural diagram of an image super-resolution reconstruction device 11 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0136] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0137] See Figure 1 , Figure 1 1 is a schematic diagram of the architecture of an image super-resolution reconstruction system provided in an embodiment of the present application, the system includes a terminal device 101 and an image super-resolution image reconstruction (SR) device 102. The embodiment of the present application does not strictly limit the number of terminal devices 101. Figure 1 The quantities shown are examples only.

[0138] The terminal device 101 is an electronic device with both data processing and data transmission and reception capabilities, and is used to provide an operation interface to the user, so that the user can select or import a low-resolution (LR) picture that requires super-resolution reconstruction. The terminal device 101 can send an LR picture to the SR device 102 and receive a high-resolution (HR) picture corresponding to the LR picture sent by the SR device 102. It should be noted that when displayed on the same device, the more pixels contained in the picture, the higher the resolution of the picture. In the embodiment of the present application, the resolution value of the picture is not strictly limited.

[0139] Optionally, the terminal device 101 can be an independent device such as a handheld terminal, a desktop terminal, a wearable device, or a component contained in an independent device (such as a chip or an integrated circuit). When the terminal device 101 is a handheld terminal, it can be a mobile phone, a tablet computer, a computer (such as a laptop computer, a PDA, etc.), etc.

[0140] The SR device 102 is a device with computing power and storage space. The storage space stores an image reconstruction model and related algorithms. The image reconstruction model includes a first reconstruction network, a SAM model, and a second reconstruction network. The SR device 102 can receive the LR image sent by the terminal device 101, perform a secondary reconstruction of the LR image using the image reconstruction model and related algorithms, and output the HR image corresponding to the LR image to the terminal device 101. It should be noted that compared to the image obtained by only upsampling processing technology, the HR image output by the SR device 102 can enhance the original boundary information and texture details of the LR image, resulting in a more refined image super-resolution reconstruction result.

[0141] Optionally, the image reconstruction model in the storage space of the SR device 102 can be a model obtained by the SR device 102 by training multiple LR sample images, or a pre-trained model directly obtained from other model training devices. The LR sample image is an image obtained by degrading the target HR sample image.

[0142] Optionally, the SR device 102 may be a cloud-based device, such as a single service in the cloud or a server cluster consisting of multiple servers, or a local device, such as a single service in the local area or a server cluster consisting of multiple servers.

[0143] The SR device 102 and the terminal device 101 may be directly connected via wired communication, such as tangible media such as metal wires and optical fibers, or may be indirectly connected via wireless communication, such as intangible media such as wireless LAN and Bluetooth.

[0144] Optionally, if the terminal device 101 has the same computing power and storage space as the SR device 102 , it can perform the aforementioned related computing operations performed by the SR device 102 .

[0145] Figure 1 In the system architecture shown, the SR device can use the image reconstruction model to perform secondary reconstruction of the LR image, so that users can obtain more detailed HR images.

[0146] See Figure 2 , Figure 2This is a flow chart of an image super-resolution reconstruction method provided by an embodiment of the present application. The method can be based on Figure 1 The system architecture shown is implemented, and the method includes but is not limited to the following steps:

[0147] Step S201: The SR device processes the LR image through a first reconstruction network to generate a first HR image.

[0148] The SR device can be Figure 1 The SR device 102 shown in the figure may also be other devices with computing capabilities; the LR image may be a terminal device (which may be Figure 1 The terminal device 101 shown, which may also be other devices with data transmission and reception capabilities, responds to user operation instructions to send images. The SR device stores an image reconstruction model, which includes a first reconstruction network, a SAM model, and a second reconstruction network.

[0149] Specifically, the first reconstruction network includes a first convolutional layer, a first residual network, an upsampling layer, and a second convolutional layer. For easier understanding, please refer to Figure 3 , Figure 3 This is a flowchart of a network reconstruction process provided by an embodiment of the present application.

[0150] First, the SR device inputs the LR image into the first convolutional layer to extract primary features of the LR image and generate eighth feature data.

[0151] Each image is composed of multiple pixels, each of which has a corresponding total grayscale value. Each total grayscale value is the sum of the grayscale values ​​of the three red, green, and blue (RGB) channels. Therefore, each pixel in the LR image has grayscale values ​​corresponding to the three RGB channels. The dimensions of the LR image are H0×W0×3, meaning it is composed of three matrices of dimension H0×W0 (i.e., matrix superposition). H0 is the number of pixels in the width direction of the LR image, W0 is the number of pixels in the length direction of the LR image, and 3 represents the number of channels in the LR image. Therefore, the resolution of the LR image can be represented by H0×W0.

[0152] The first convolution layer contains multiple convolution kernels. The dimension of the first convolution layer is F×F×C×N. The first convolution layer includes N convolution kernels. The dimension of each convolution kernel is F×F×C, that is, it is composed of C matrices with a dimension of F×F. The number of matrix layers C of each convolution kernel is the same as the number of matrix layers (i.e., the number of channels) of the input data of the first convolution layer (i.e., the LR image). During the convolution process, each layer matrix of each convolution kernel in the first convolution layer is convolved with each layer matrix of the LR image in a one-to-one correspondence. Finally, the dimension of the output data of the first convolution layer (i.e., the eighth feature data) is H. ′×W ′ ×N. Specifically, the dimension of each layer matrix of the eighth feature data (H ′ ×W ′ ) depends on the step size S of the convolution kernel during convolution calculation and the dimension (H0×W0) of each layer matrix of the LR image. The number of matrix layers of the eighth feature data is equal to the number of convolution kernels N. The relevant calculation formula is as follows:

[0153]

[0154] For example, if the dimension of the LR image is 6×6×3, the dimension of the first convolution layer is 3×3×3×4, and the stride of each convolution kernel is 1, then the dimension of the output eighth feature data is 4×4×4. For easier understanding, please refer to Figure 4 , Figure 4 This is a schematic diagram of a convolutional layer algorithm provided in an embodiment of the present application. Specifically, the LR image is convolved with the four convolution kernels in the first convolution layer to obtain four matrices of dimension 4×4, which are then concatenated to obtain the eighth feature data of dimension 4×4×4. Symbol It is used to represent matrix concatenation, and the symbol ⊕ is used to represent matrix fusion (i.e. matrix addition).

[0155] The following uses convolution kernel 401 as an example to illustrate the relevant algorithm. The three matrices of 3×3 dimensions in convolution kernel 401 correspond one-to-one with the three 6×6 matrices in the LR image to perform dot product calculations. During the calculation process, each 3×3 matrix needs to traverse each element in the corresponding 6×6 matrix, and the distance moved during the traversal is the step size. That is, matrix A in convolution kernel 401 traverses each element in matrix B in the corresponding LR image by moving one grid at a time, generating matrix C. Each element in matrix C contains the fusion information of multiple elements in matrix B of the LR image. Finally, the LR image is processed by convolution kernel 401 to obtain three matrices of 4×4 dimensions, which are fused to generate a matrix of 4×4 dimensions. It is not difficult to understand that the eighth feature data contains the weighted fusion information of the elements in the LR image.

[0156] It should be noted that the algorithm of the second convolutional layer in the first reconstruction network is consistent with that of the first convolutional layer. The dimension of the second convolutional layer and the step size of the convolution kernel can be set according to actual application requirements, and the embodiment of the present application does not limit this.

[0157] Furthermore, the SR device inputs the eighth feature data into a first residual network, and fuses the output data of the first residual network with the LR image to generate first feature data, the dimension of the first feature data being the same as the dimension of the LR image. The first residual network includes N residual blocks, and the N residual blocks are used to perform iterative feature extraction processing on the input eighth feature data, where N is an integer greater than 1. For ease of understanding, please refer to Figure 5 , Figure 5 A schematic diagram of a residual block algorithm provided in an embodiment of the present application.

[0158] Specifically, the first residual block includes two convolutional layers and two rectified linear unit (ReLU) activation layers. The input feature data F of the first residual block is IN The convolution layer 501, the ReLU activation layer 502, the convolution layer 503, and the ReLU activation layer 504 are processed in sequence. The convolution layer 501 and the convolution layer 503 are used to process the input feature data F IN Perform two feature extractions, and the ReLU activation layer 502 and the ReLU activation layer 504 use the ReLU function to avoid gradient explosion and gradient disappearance problems in the calculation process. The output feature data F of the first residual block OUT The output feature data of the ReLU activation layer 504 and the input feature data F IN The fused data is used as the input feature data of the next residual block. It should be noted that the first residual block is any one of the N residual blocks. It is not difficult to understand that the first residual network is used to perform deep feature extraction on the eighth feature data.

[0159] Therefore, the first feature information generated by processing the LR image through the first convolutional layer and the first residual network fuses the information of the LR image and the rich feature information in the LR image.

[0160] Next, the SR device performs feature restoration on the first feature data through the upsampling layer and the second convolution layer to generate a first high-resolution HR image.

[0161] Specifically, the upsampling layer is used to enlarge the dimension of the first feature data to the target size to generate the tenth feature data. The dimension of the tenth feature data is KH0×KW0×3, and it should be noted that K is the magnification factor. Optionally, the algorithm used by the upsampling layer can be an interpolation method, a deconvolution method, or a depooling method. The second convolution layer is used to restore the features of the elements in the tenth feature data to generate a first HR image with a dimension of KH0×KW0×3. Therefore, the resolution of the first HR image is KH0×KW0.

[0162] That is to say, the SR device processes the LR image through the first reconstruction network to improve the resolution of the LR image.

[0163] Considering that the first HR image will weaken the original boundary information and texture details in the LR image due to the rapid conversion from low resolution to high resolution, the SR device further continues to perform the following steps S202 and S203 on the first HR image.

[0164] Step S202: The SR device processes the first HR image through a supervised attention module (SAM) model to generate second feature data.

[0165] Specifically, the SAM model includes the seventh convolutional layer, the eighth convolutional layer, and the S-shaped growth curve Sigmoid activation layer. For easier understanding, please refer to Figure 6 , Figure 6 This is a flow chart of a supervised attention model provided in an embodiment of the present application.

[0166] First, the SR device extracts features from the first HR image through the seventh convolutional layer to generate ninth feature data with a dimension of KH0×KW0×3, and concatenates the tenth feature data generated by the upsampling layer in the first reconstruction network with the first HR image to generate eleventh feature data with a dimension of KH0×KW0×6.

[0167] Next, the SR device performs feature calibration processing on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data with a dimension of KH0×KW0×3. Specifically, the eighth convolutional layer is used to reduce the dimension of the eleventh feature data to the dimension of the twelfth feature data, and the Sigmoid activation layer uses a Sigmoid function to perform attention calculation on the feature data obtained by the eighth convolutional layer. Therefore, the twelfth feature data includes attention scores of multiple pixel-level features, representing the importance of multiple pixel-level features to the reconstruction result.

[0168] It should be noted that the calculation formula of the Sigmoid function is as follows:

[0169] S(x)=1 / (1+e -x )

[0170] Among them, x is the element value in the feature data obtained by processing the eighth convolutional layer, S(x) is the attention score corresponding to the element value x, and the larger S(x) is, the more important the element is to the reconstruction result.

[0171] Furthermore, the SR device multiplies the twelfth feature data with the ninth feature data (matrix dot product) to generate thirteenth feature data of dimension KH0×KW0×3, and then fuses the thirteenth feature data with the first HR image to generate second feature data of dimension KH0×KW0×3.

[0172] Therefore, the SAM model performs attention score weighted calculation on the first HR image, which can enhance the important features in the first HR image and generate second feature data.

[0173] It should be noted that the algorithms of the seventh convolutional layer and the eighth convolutional layer in the SAM model are consistent with the first convolutional layer of the first reconstruction network in step S201. The dimensions of each convolutional layer and the step size of the convolution kernel can be set according to actual application requirements, and the embodiments of this application do not limit this.

[0174] Step S203: The SR device processes the first HR image and the second feature data through a second reconstruction network to output a second HR image.

[0175] Specifically, the second reconstruction network includes the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer and the second residual network. For easier understanding, please refer to Figure 7 , Figure 7 This is another flowchart of network reconstruction provided in an embodiment of the present application.

[0176] The SR device performs feature extraction on the first HR image through the third convolutional layer to generate third feature data with a dimension of KH0×KW0×3, and then concatenates the third feature data with the second feature data to generate fourth feature data with a dimension of KH0×KW0×6.

[0177] Next, the SR device performs feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network to generate fifth feature data of dimension KH0×KW0×3. Specifically, the fourth convolutional layer is used to reduce the dimension of the fourth feature data to generate fourteenth feature data of dimension KH0×KW0×3. The second residual network includes M connected residual blocks, and the M residual blocks are used to perform iterative feature extraction on the fourteenth feature data (M is an integer greater than 1) for deep feature extraction of the fourteenth feature data. The SR device fuses the output feature data of the second residual network with the fourteenth feature data to generate the fifth feature data.

[0178] Furthermore, the SR device extracts features from the fifth feature data through the fifth convolutional layer to generate sixth feature data of dimension KH0×KW0×3, and then concatenates the sixth feature data with the second feature data to generate seventh feature data of KH0×KW0×6. Finally, the seventh feature data is restored through the sixth convolutional layer to output a second HR image of dimension KH0×KW0×3, and the resolution of the second HR image is KH0×KW0.

[0179] Therefore, the SR device can perform two splicing processes on the first HR image and the second feature data through the second reconstruction network. This not only preserves the features of the LR image extracted by the first reconstruction network, but also allows the second reconstruction network to focus on the features in the LR image that are more important to the reconstruction effect, thereby outputting a second HR image with enhanced boundary information and texture details.

[0180] It should be noted that the algorithm of the second residual network in the second reconstruction network is consistent with the first residual network of the first reconstruction network in step S201, and the algorithms of the third to fifth convolutional layers in the second reconstruction network are consistent with the first convolutional layer of the first reconstruction network in step S201. The dimension of each convolutional layer and the step size of the convolution kernel can be set according to actual application requirements, and the embodiment of the present application does not limit this.

[0181] In the embodiment of the present application, the image reconstruction model in the SR device in the above steps S201-S203 can be a model obtained by the SR device by training multiple LR sample images (i.e., batch training), see Figure 8 , Figure 8 This is a flowchart of a training method for an image reconstruction model provided in an embodiment of the present application.

[0182] S801: The SR device processes the LR sample image through a first reconstruction network to generate a first HR sample image.

[0183] It should be noted that the SR device stores an image reconstruction model, which includes a first reconstruction network, a SAM model, and a second reconstruction network. The LR sample image is an image obtained by degrading the target HR sample image. The embodiment of the present application does not limit the degradation technology, and the degradation technology can be a model composed of one or more of downsampling, blurring, noise, and random permutation methods.

[0184] Specifically, the first reconstruction network includes a first convolutional layer, a first residual network, an upsampling layer and a second convolutional layer.

[0185] The SR device inputs the LR sample image into the first convolution layer to extract primary features of the LR sample image and generate eighth feature data. It should be noted that the algorithm of the second convolution layer in the first reconstruction network is consistent with that of the first convolution layer.

[0186] Then, the eighth feature data is input into the first residual network, and the output feature data of the first residual network is fused with the LR sample image to generate the first feature data. The first residual network includes N connected residual blocks, and the N residual blocks are used to perform iterative feature extraction processing on the input eighth feature data, where N is an integer greater than 1. For easier understanding, please refer to Figure 9 , Figure 9 A schematic diagram of another residual block algorithm provided in an embodiment of the present application.

[0187] Specifically, the first residual network includes N connected residual blocks, wherein the first residual block includes two convolutional layers, an instance normalization (IN) layer and two ReLU activation layers. The input feature data F of the first residual block is IN The convolution layer 901, IN layer 902, ReLU activation layer 903, convolution layer 904, and ReLU activation layer 905 are sequentially processed. The convolution layer 901 and the convolution layer 904 are used to process the input feature data F IN Perform feature extraction twice, and the IN layer 902 is used to extract the output feature data F of the convolution layer 901. MID Half of the feature data F MID1 Perform instance normalization processing, the ReLU activation layer 903 and the ReLU activation layer 905 are used to use the ReLU function to avoid gradient explosion and gradient disappearance problems in the calculation process. The input feature data of the ReLU activation layer 903 is the output feature data F of the convolution layer 901 MID The other half of the feature data F MID2 The concatenated data of the IN layer output feature data and the output feature data F of the first residual block are OUT The output feature data of the ReLU activation layer 905 and the input feature data F IN The fused data is used as the input feature data of the next residual block. It should be noted that the first residual block is any one of the N residual blocks. It is not difficult to understand that the first residual network is used to perform deep feature extraction on the eighth feature data.

[0188] It should be noted that the input feature data of each residual block in the first residual network is the output feature data of the previous unit. When the parameters of the previous unit change, the distribution of the input feature data will change significantly, and the feature information in the original input feature data will be easily lost, and the convergence speed during model training will be reduced. To solve this problem, each residual block in the first residual network is provided with an IN layer, which is used to normalize half of the feature data of the input feature data obtained by the first convolutional layer. That is to say, each residual block of the first residual network performs half-instance normalization on the input feature data. This can not only retain the feature information in the original input feature data and improve the super-resolution reconstruction effect of the image reconstruction model, but also speed up the convergence speed of the image reconstruction model and improve the training speed of the image reconstruction model compared to batch normalization or full instance normalization.

[0189] When the image reconstruction model training in the SR device is completed, that is, the parameter optimization in the image reconstruction model is completed, the SR device can delete the IN layer of each residual block in the first residual network to simplify the model. At this time, the first residual network is equivalent to the first residual network in step S201.

[0190] Furthermore, the SR device performs feature restoration on the first feature data through the upsampling layer and the second convolution layer to generate a first high-resolution HR sample image.

[0191] S802: The SR device processes the first HR sample image through the SAM model to generate second feature data.

[0192] Specifically, the SAM model includes a seventh convolutional layer, an eighth convolutional layer, and an S-shaped growth curve Sigmoid activation layer.

[0193] First, the SR device extracts features from the first HR sample image through the seventh convolutional layer to generate ninth feature data, and concatenates the tenth feature data generated by the upsampling layer in the first reconstruction network with the first HR sample image to generate eleventh feature data.

[0194] Next, the SR device performs feature calibration processing on the eleventh feature data through the eighth convolution layer and the Sigmoid activation layer. Specifically, the eighth convolution layer is used to reduce the dimension of the eleventh feature data to the dimension of the twelfth feature data, and the Sigmoid activation layer uses the Sigmoid function to perform attention calculation on the feature data processed by the eighth convolution layer. Therefore, the twelfth feature data includes attention scores of multiple pixel-level features, representing the importance of multiple pixel-level features to the reconstruction result.

[0195] Furthermore, the SR device multiplies the twelfth feature data with the ninth feature data (matrix dot product) to generate thirteenth feature data, and then fuses the thirteenth feature data with the first HR sample image to generate second feature data.

[0196] Therefore, the SAM model performs attention score weighted calculation on the first HR sample image, which can enhance the important features in the first HR sample image and generate second feature data.

[0197] It should be noted that the algorithms of the seventh convolutional layer and the eighth convolutional layer in the SAM model are consistent with the first convolutional layer of the first reconstruction network in step S801.

[0198] S803: The SR device processes the first HR sample image and the second feature data through a second reconstruction network, and outputs a second HR sample image.

[0199] Specifically, the second reconstruction network includes a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer and a second residual network.

[0200] The SR device performs feature extraction on the first HR sample image through the third convolutional layer to generate third feature data, and then concatenates the third feature data with the second feature data to generate fourth feature data.

[0201] Next, the SR device extracts features from the fourth feature data through the fourth convolutional layer and the second residual network to generate fifth feature data. Specifically, the fourth convolutional layer is used to reduce the dimension of the fourth feature data to generate fourteenth feature data, and the second residual network includes M connected residual blocks, which are used to perform iterative feature extraction on the fourteenth feature data (M is an integer greater than 1) and perform deep feature extraction on the fourteenth feature data; the SR device fuses the output feature data of the second residual network with the fourteenth feature data to generate the fifth feature data.

[0202] Furthermore, the SR device extracts features from the fifth feature data through the fifth convolutional layer to generate sixth feature data, concatenates the sixth feature data with the second feature data to generate seventh feature data, and finally restores features from the seventh feature data through the sixth convolutional layer to output a second HR sample image.

[0203] It should be noted that the algorithm of the second residual network in the second reconstruction network is consistent with the first residual network of the first reconstruction network in step S801, and the algorithms of the third to fifth convolutional layers in the second reconstruction network are consistent with the first convolutional layer of the first reconstruction network in step S801.

[0204] In practical applications, in order to better optimize the image reconstruction model involved in the above steps S801-S803, the SR device often uses a large number of LR sample images for model training, and each LR sample image in the multiple LR sample images needs to perform the same operations as the above steps S801-S803.

[0205] S804: The SR device optimizes parameters in the first reconstruction network, the SAM model, and the second reconstruction network according to the first HR sample image, the second HR sample image, and the target HR sample image.

[0206] Specifically, the SR device calculates the difference value L between the first HR sample image, the second HR sample image and the target HR sample image through a loss function, and the loss function is as follows:

[0207]

[0208] Specifically, Y is the number of LR sample images, is the first HR sample image corresponding to the i-th LR sample image, is the second HR sample image corresponding to the i-th LR sample image, is the target HR sample image corresponding to the i-th LR sample image, where i is equal to 1, 2, …, Y-1, Y.

[0209] The SR device continuously optimizes parameters in the first reconstruction network, the SAM model, and the second reconstruction network in the image reconstruction model based on the gap value L, updates gradients, and performs parameter backpropagation until the gap value L is less than a preset threshold. Optionally, the preset threshold may be a default threshold set by the SR device, or a threshold that is modified based on the resolution and fineness requirements of the first HR sample image and the second HR sample image.

[0210] Finally, the SR device saves the trained image reconstruction model to facilitate subsequent super-resolution reconstruction of the LR image to be reconstructed.

[0211] In addition, in an embodiment of the present application, the multiple models in the SR device in the above steps S801-S804 can be pre-trained models directly obtained from other model training devices, and the training steps performed by the model training device are based on the same principles as the above steps S801-S804.

[0212] Through the embodiment of the present application, the SR device can receive the LR image sent by the terminal device, and perform secondary reconstruction of the LR image in combination with the first reconstruction network, SAM model and second reconstruction model in the image reconstruction model. It can retain the features of the LR image extracted by the first reconstruction network, and can also enable the second reconstruction network to focus on the features in the LR image that are more important to the reconstruction effect, thereby outputting a second HR image with enhanced boundary information and texture details, that is, outputting a fine image super-resolution reconstruction result.

[0213] The above describes in detail the method of the embodiment of the present application. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0214] It can be understood that the device provided in the embodiments of the present application, such as an image super-resolution reconstruction device based on an image reconstruction model, includes hardware structures, software modules, or a combination of hardware structures and software structures corresponding to executing each function in order to realize the functions in the above method embodiments.

[0215] Those skilled in the art should easily appreciate that, in combination with the units and steps of the various examples described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different device implementations to implement the aforementioned method embodiments in different usage scenarios, and different implementations of the devices should not be considered to exceed the scope of the embodiments of the present application.

[0216] In the embodiments of the present application, the device can be divided into functional modules. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one functional module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. For example, taking the case of dividing the functional modules of the device by integration as an example, the present application cites several possible processing devices.

[0217] See Figure 10 , Figure 10 is a structural diagram of an image super-resolution reconstruction device provided in an embodiment of the present application. The image super-resolution reconstruction device 10 can be Figure 1The SR device 102 in the illustrated embodiment, or a device in the SR device 102; the image super-resolution reconstruction apparatus 10 may include a first processing unit 1001, a second processing unit 1002, a third processing unit 1003, a fourth processing unit 1004, a first stitching unit 1005, a fifth processing unit 1006, a sixth processing unit 1007, a second stitching unit 1008, and an output unit 1009, wherein each unit is described in detail as follows:

[0218] A first processing unit 1001 is configured to perform feature extraction on the low-resolution LR image using a first convolutional layer and a first residual network in a first reconstruction network to generate first feature data, wherein each pixel in the LR image has a grayscale value corresponding to three channels (red, green, and blue, RGB);

[0219] A second processing unit 1002 is configured to perform feature restoration on the first feature data through an upsampling layer and a second convolutional layer in the first reconstruction network to generate a first high-resolution HR image;

[0220] The third processing unit 1003 is configured to perform feature calibration on the first high-resolution image using a SAM model to generate second feature data;

[0221] a fourth processing unit 1004, configured to perform feature extraction on the first HR image through a third convolutional layer in a second reconstruction network to generate third feature data;

[0222] A first splicing unit 1005 is configured to splice the third feature data with the second feature data to generate fourth feature data;

[0223] A fifth processing unit 1006 is configured to perform feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data;

[0224] a sixth processing unit 1007, configured to perform feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data;

[0225] A second splicing unit 1008 is configured to splice the sixth feature data with the second feature data to generate seventh feature data;

[0226] The output unit 1009 is configured to perform feature restoration on the seventh feature data through the sixth convolutional layer in the second reconstruction network, and output a second HR image.

[0227] In a possible implementation, in extracting features from the LR image using the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data, the first processing unit 1001 is specifically configured to:

[0228] Performing feature extraction on the LR image through the first convolutional layer to generate eighth feature data;

[0229] Inputting the eighth feature data into the first residual network, wherein the first residual network includes N residual blocks, and the N residual blocks are used to perform iterative feature extraction on the input eighth feature data, where N is an integer greater than 1;

[0230] The output data of the first residual network is fused with the LR image to generate the first feature data.

[0231] In another possible implementation, the SAM model includes a seventh convolutional layer, an eighth convolutional layer, and a Sigmoid activation layer; and in performing feature calibration on the first HR image using the SAM model to generate second feature data, the third processing unit 1003 is specifically configured to:

[0232] performing feature extraction on the first HR image through the seventh convolutional layer to generate ninth feature data;

[0233] splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first HR image to generate eleventh feature data;

[0234] Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data;

[0235] multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data;

[0236] The thirteenth feature data is fused with the first HR image to generate the second feature data.

[0237] In yet another possible implementation, in the aspect of extracting features from the fourth feature data using the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data, the fifth processing unit 1006 is specifically configured to:

[0238] performing feature extraction on the fourth feature data through the fourth convolutional layer to generate fourteenth feature data;

[0239] Inputting the fourteenth feature data into the second residual network, wherein the second residual network includes M residual blocks, and the M residual blocks are used to perform iterative feature extraction on the input fourteenth feature data, where M is an integer greater than 1;

[0240] The output data of the second residual network is fused with the fourteenth feature data to generate the fifth feature data.

[0241] It should be noted that, in the embodiments of the present application, the specific implementation and technical effects of each unit can also refer to Figures 1-9 The corresponding description of the illustrated embodiment.

[0242] See Figure 11 , Figure 11 An embodiment of the present application provides an image super-resolution reconstruction device 11, which includes a processor 1101, a memory 1102 and a communication interface 1103, wherein the processor 1101, the memory 1102 and the communication interface 1103 are interconnected via a bus.

[0243] The processor 1101 may be one or more central processing units (CPUs). When the processor 1101 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0244] The memory 1102 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used for related computer programs and data.

[0245] Communication interface 1103 is used to receive and send data. Optionally, the communication interface 1103 receives a LR image sent by a terminal device and sends the LR image to the processor 1101. Optionally, the communication interface 1103 receives a second HR image corresponding to the LR image sent by the processor 1101 and sends the second HR image to the terminal device.

[0246] The processor 1101 is configured to read the computer program code stored in the memory 1102 and perform the following operations:

[0247] Performing feature extraction on the low-resolution LR image through the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data, wherein each pixel in the LR image has a grayscale value corresponding to the three channels of red, green, and blue (RGB);

[0248] Performing feature restoration on the first feature data through the upsampling layer and the second convolutional layer in the first reconstruction network to generate a first high-resolution HR image;

[0249] Performing feature calibration on the first HR image using a SAM model to generate second feature data;

[0250] performing feature extraction on the first HR image through a third convolutional layer in a second reconstruction network to generate third feature data;

[0251] concatenating the third feature data with the second feature data to generate fourth feature data;

[0252] performing feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data;

[0253] performing feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data;

[0254] splicing the sixth feature data with the second feature data to generate seventh feature data;

[0255] The sixth convolutional layer in the second reconstruction network is used to perform feature restoration on the seventh feature data, and a second HR image is output.

[0256] In a possible implementation, in the aspect of extracting features from the LR image through the first convolutional layer and the first residual network in the first reconstruction network to generate the first feature data, the processor 1101 is specifically configured to:

[0257] Performing feature extraction on the LR image through the first convolutional layer to generate eighth feature data;

[0258] Inputting the eighth feature data into the first residual network, wherein the first residual network includes N residual blocks, and the N residual blocks are used to perform iterative feature extraction on the input eighth feature data, where N is an integer greater than 1;

[0259] The output data of the first residual network is fused with the LR image to generate the first feature data.

[0260] In another possible implementation, the SAM model includes a seventh convolutional layer, an eighth convolutional layer, and a Sigmoid activation layer; and in performing feature calibration on the first HR image using the SAM model to generate the second feature data, the processor 1101 is specifically configured to:

[0261] performing feature extraction on the first HR image through the seventh convolutional layer to generate ninth feature data;

[0262] splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first HR image to generate eleventh feature data;

[0263] Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data;

[0264] multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data;

[0265] The thirteenth feature data is fused with the first HR image to generate the second feature data.

[0266] In yet another possible implementation, in the aspect of extracting features from the fourth feature data using the fourth convolutional layer and the second residual network in the second reconstruction network to generate the fifth feature data, the processor 1101 is specifically configured to:

[0267] performing feature extraction on the fourth feature data through the fourth convolutional layer to generate fourteenth feature data;

[0268] Inputting the fourteenth feature data into the second residual network, wherein the second residual network includes M residual blocks, and the M residual blocks are used to perform iterative feature extraction on the input fourteenth feature data, where M is an integer greater than 1;

[0269] The output data of the second residual network is fused with the fourteenth feature data to generate the fifth feature data.

[0270] It should be noted that the implementation of each operation can also refer to Figures 1-9 The corresponding description of the illustrated embodiment.

[0271] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a network device, Figure 2 or Figure 8 The method flow shown is realized.

[0272] The "multiple" mentioned in the embodiments of this application refers to two or more, and, unless otherwise specified, the "first" in the first reconstruction network, first convolutional layer, first residual network, first residual block, first feature data, first HR image, first processing unit, and first splicing unit mentioned in the embodiments of this application is only used as a name identifier and is not used to limit the order, timing, priority, or importance of multiple objects. The same rule applies to "second," "third," and "fourth," etc.

[0273] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image super-resolution reconstruction method based on an image reconstruction model, characterized in that: The image reconstruction model includes a first reconstruction network, a supervised attention SAM model and a second reconstruction network, and the method includes: Performing feature extraction on the low-resolution LR image through the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data, wherein each pixel in the LR image has a grayscale value corresponding to three channels (red, green, and blue, RGB); Performing feature restoration on the first feature data through the upsampling layer and the second convolutional layer in the first reconstruction network to generate a first high-resolution HR image; Performing feature calibration on the first high-resolution HR image using the SAM model to generate second feature data; performing feature extraction on the first high-resolution HR image through a third convolutional layer in the second reconstruction network to generate third feature data; concatenating the third feature data with the second feature data to generate fourth feature data; performing feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data; performing feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data; splicing the sixth feature data with the second feature data to generate seventh feature data; Performing feature restoration on the seventh feature data through the sixth convolutional layer in the second reconstruction network, and outputting a second HR image; The SAM model includes a seventh convolutional layer, an eighth convolutional layer, and an S-shaped growth curve Sigmoid activation layer; and the step of performing feature calibration on the first high-resolution HR image using the SAM model to generate second feature data includes: performing feature extraction on the first high-resolution HR image through the seventh convolutional layer to generate ninth feature data; splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first high-resolution HR image to generate eleventh feature data; Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data; multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data; The thirteenth feature data is fused with the first high-resolution HR image to generate the second feature data.

2. The method according to claim 1, characterized in that The step of extracting features from the LR image through the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data includes: Performing feature extraction on the LR image through the first convolutional layer to generate eighth feature data; Inputting the eighth feature data into the first residual network, wherein the first residual network includes N residual blocks, and the N residual blocks are used to perform iterative feature extraction on the input eighth feature data, where N is an integer greater than 1; The output data of the first residual network is fused with the LR image to generate the first feature data.

3. The method according to claim 1 or 2, characterized in that The step of extracting features from the fourth feature data using the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data includes: performing feature extraction on the fourth feature data through the fourth convolutional layer to generate fourteenth feature data; Inputting the fourteenth feature data into the second residual network, wherein the second residual network includes M residual blocks, and the M residual blocks are used to perform iterative feature extraction on the input fourteenth feature data, where M is an integer greater than 1; The output data of the second residual network is fused with the fourteenth feature data to generate the fifth feature data.

4. A training method for an image reconstruction model, characterized in that: The image reconstruction model includes a first reconstruction network, a supervised attention SAM model and a second reconstruction network, and the method includes: Performing feature extraction on the low-resolution (LR) sample image using the first convolutional layer and the first residual network in the first reconstruction network to generate first feature data, wherein each pixel in the LR sample image has grayscale values ​​corresponding to three channels (red, green, and blue, RGB), and the LR sample image is an image obtained by degrading the target high-resolution (HR) sample image; Perform feature restoration on the first feature data through the upsampling layer and the second convolution layer in the first reconstruction network to generate a first HR sample image; Performing feature calibration on the first HR sample image using the SAM model to generate second feature data; performing feature extraction on the first HR sample image through the third convolutional layer in the second reconstruction network to generate third feature data; concatenating the third feature data with the second feature data to generate fourth feature data; performing feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data; performing feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data; splicing the sixth feature data with the second feature data to generate seventh feature data; Performing feature restoration on the seventh feature data through the sixth convolutional layer in the second reconstruction network, and outputting a second HR sample image; Optimizing parameters in the first reconstruction network, the SAM model, and the second reconstruction network according to the first HR sample image, the second HR sample image, and the target high-resolution HR sample image; The SAM model includes a seventh convolutional layer, an eighth convolutional layer, and an S-shaped growth curve Sigmoid activation layer; and the step of performing feature calibration on the first high-resolution HR sample image using the SAM model to generate second feature data includes: performing feature extraction on the first high-resolution HR sample image through the seventh convolutional layer to generate ninth feature data; splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first high-resolution HR sample image to generate eleventh feature data; Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data; multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data; The thirteenth feature data is fused with the first high-resolution HR sample image to generate the second feature data.

5. The method according to claim 4, characterized in that: The first residual network includes N connected residual blocks, wherein the first residual block includes two convolutional layers, an instance normalization (IN) layer, and two rectified linear unit (ReLU) activation layers, the input data of the IN layer includes half of the feature data of the output data of the first convolutional layer, and the input data of the first ReLU activation layer includes the concatenation of the other half of the feature data of the output data of the first convolutional layer and the output data of the IN layer; the output data of the first residual block includes the fusion data of the output data of the second ReLU activation layer in the first residual block and the input data of the first residual block, which is used as the input data of the next residual block; the first residual block is any one of the N residual blocks, where N is an integer greater than 1; The second residual network includes M connected residual blocks, wherein the second residual block includes two convolutional layers, an instance normalization IN layer and two rectified linear unit ReLU activation layers, the input data of the IN layer includes half of the feature data of the output data of the first convolutional layer, and the input data of the first ReLU activation layer includes the concatenation of the other half of the feature data of the output data of the first convolutional layer and the output data of the IN layer; the output data of the second residual block includes the fusion data of the output data of the second ReLU activation layer in the second residual block and the input data of the second residual block, which is used as the input data of the next residual block; the second residual block is any one of the M residual blocks, and M is an integer greater than 1.

6. The method according to claim 4, characterized in that Optimizing parameters in the first reconstruction network, the SAM model, and the second reconstruction network according to the first HR sample image, the second HR sample image, and the target high-resolution HR sample image includes: Calculating the difference between the first HR sample image, the second HR sample image and the target high-resolution HR sample image by using a loss function; Optimizing parameters in the first reconstruction network, the SAM model, and the second reconstruction network until the gap value is smaller than a preset threshold.

7. An image super-resolution reconstruction device, characterized in that: The device comprises: A first processing unit is configured to perform feature extraction on the low-resolution LR image through a first convolutional layer and a first residual network in a first reconstruction network to generate first feature data, wherein each pixel in the LR image has a grayscale value corresponding to three channels (red, green, and blue (RGB)). a second processing unit, configured to perform feature restoration on the first feature data through an upsampling layer and a second convolutional layer in the first reconstruction network to generate a first high-resolution HR image; a third processing unit, configured to perform feature calibration on the first high-resolution HR image using a SAM model to generate second feature data; a fourth processing unit, configured to perform feature extraction on the first high-resolution HR image through a third convolutional layer in the second reconstruction network to generate third feature data; a first splicing unit, configured to splice the third feature data with the second feature data to generate fourth feature data; a fifth processing unit, configured to perform feature extraction on the fourth feature data through the fourth convolutional layer and the second residual network in the second reconstruction network to generate fifth feature data; a sixth processing unit, configured to perform feature extraction on the fifth feature data through a fifth convolutional layer in the second reconstruction network to generate sixth feature data; a second splicing unit, configured to splice the sixth feature data with the second feature data to generate seventh feature data; an output unit, configured to perform feature restoration on the seventh feature data through the sixth convolutional layer in the second reconstruction network, and output a second HR image; The SAM model includes a seventh convolutional layer, an eighth convolutional layer, and an S-shaped growth curve Sigmoid activation layer; in performing feature calibration on the first high-resolution HR image through the SAM model to generate second feature data, the third processing unit is specifically used to: performing feature extraction on the first high-resolution HR image through the seventh convolutional layer to generate ninth feature data; splicing tenth feature data generated by processing the ninth feature data through the upsampling layer with the first high-resolution HR image to generate eleventh feature data; Performing feature calibration on the eleventh feature data through the eighth convolutional layer and the Sigmoid activation layer to generate twelfth feature data; multiplying the twelfth characteristic data by the ninth characteristic data to generate thirteenth characteristic data; The thirteenth feature data is fused with the first high-resolution HR image to generate the second feature data.

8. An image super-resolution reconstruction device, characterized in that: It includes a processor, a memory, and a communication interface, wherein the communication interface is used to perform receiving and / or sending operations under the control of the processor, the memory is used to store a computer program, and the processor is used to call the computer program to implement the method described in any one of claims 1 to 3.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed on a processor, implements the method according to any one of claims 1 to 3.

Images