A super-resolution image reconstruction method, device, medium and equipment

Abstract

The present invention discloses a super-resolution image reconstruction method, device, medium and equipment, the method comprising: upsampling a low-resolution image to be reconstructed to obtain an upsampled image; inputting the low-resolution image to be reconstructed into a preset shallow feature expression extraction module to obtain image shallow features; inputting the image shallow features into a preset multi-branch feature extraction module to obtain image global features; inputting the image global features and the image shallow features into a preset attention fusion network module to obtain image fusion features; inputting the image fusion features and the upsampled image into a preset upsampling network module to obtain a reconstructed super-resolution image. The present invention can effectively retain deep and shallow information, avoid the occurrence of shallow feature loss, enhance high-contribution feature information, suppress redundant information, and improve the reconstruction quality of super-resolution images.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and in particular to a super-resolution image reconstruction method, device, medium and equipment. Background Art

[0002] In daily life, due to the limitations of image acquisition equipment hardware, the impact of the environment in which the image is acquired, and the loss of the acquired image during transmission, the final image usually has unclear image edges, texture loss, or even blurriness. In addition, the loss of important texture details may even affect the information transmitted by the image, leading to misjudgment. Therefore, improving the quality of the image to obtain high-quality and clear images has become an urgent need.

[0003] Convolutional neural network provides a new solution for image super-resolution. Increasing the depth of the network is the simplest way to improve the super-resolution performance. With the improvement of computing power, the deep learning network gradually becomes deeper, and the network model performance gradually improves. Although early researchers applied deep learning networks to greatly improve the quality of the final reconstructed image, there are still the following problems in the design of super-resolution research networks: the current deep learning network modules are single, and many methods of stacking multiple modules are used. This will cause a large amount of feature redundancy, further slowing down the calculation speed. Different features contribute differently to the reconstructed image, but the current single network model treats these features equally, and the failure to distinguish between high-contribution information and redundant information in the features during training will affect the quality of the final reconstructed image. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a super-resolution image reconstruction method, device, medium and equipment. To achieve the above purpose, the present invention is implemented by adopting the following technical solutions:

[0005] In a first aspect, the present invention provides a super-resolution image reconstruction method, comprising:

[0006] Upsampling the low-resolution image to be reconstructed to obtain an upsampled image;

[0007] Inputting the low-resolution image to be reconstructed into a preset shallow feature expression extraction module to obtain shallow features of the image;

[0008] Inputting the shallow features of the image into a preset multi-branch feature extraction module to obtain the global features of the image;

[0009] Inputting the global image features and the shallow image features into a preset attention fusion network module to obtain image fusion features; the attention fusion network module includes fusing spatial attention with pixel attention;

[0010] The image fusion feature and the upsampled image are input into a preset upsampled network module to obtain a reconstructed super-resolution image.

[0011] In combination with the first aspect, optionally, the multi-branch feature extraction module includes three sub-multi-branch feature extraction modules, each sub-multi-branch extraction module includes four links, wherein the features extracted by each link are fused with the features extracted by the adjacent links and then connected by a fully connected layer.

[0012] In combination with the first aspect, optionally, the attention fusion network module includes three sub-attention fusion modules, each sub-attention fusion module includes two branch networks, one branch network includes a feature convolution layer, and the other branch network includes an attention fusion residual module and a feature convolution layer.

[0013] In combination with the first aspect, optionally, the attention fusion residual module includes a spatial attention mechanism fusion residual module and a pixel attention mechanism fusion residual module, and the spatial attention mechanism fusion residual module and the pixel attention mechanism fusion residual module are combined in parallel.

[0014] In combination with the first aspect, optionally, the spatial attention mechanism fuses a residual module, including an expanded convolution layer, a relu activation layer, a contracted convolution layer and a sigmoid activation layer, and the pixel attention mechanism fuses a residual module, including a 1×1 convolution layer and a sigmoid activation layer.

[0015] In a second aspect, the present invention provides a super-resolution image reconstruction device, comprising:

[0016] Pre-upsampling module: upsamples the low-resolution image to be reconstructed to obtain an upsampled image;

[0017] Shallow feature extraction module: input the low-resolution image to be reconstructed into a preset shallow feature expression extraction module to obtain shallow features of the image;

[0018] Global feature extraction module: input the shallow features of the image into a preset multi-branch feature extraction module to obtain the global features of the image;

[0019] Feature fusion module: input the global image features and the shallow image features into a preset attention fusion network module to obtain image fusion features; the attention fusion network module includes fusing spatial attention with pixel attention;

[0020] Upsampling reconstruction module: input the image fusion features and the upsampled image into a preset upsampling module to obtain a reconstructed super-resolution image.

[0021] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the super-resolution image reconstruction method as described in any one of the first aspects is implemented.

[0022] In a fourth aspect, the present invention provides an electronic device, characterized in that it includes:

[0023] A memory for storing instructions;

[0024] A processor is used to execute the instructions so that the device performs operations to implement the super-resolution image reconstruction method as described in any one of the first aspects.

[0025] Compared with the prior art, the super-resolution image reconstruction method, device, medium and equipment provided by the embodiments of the present invention have the following beneficial effects:

[0026] The present invention upsamples the low-resolution image to be reconstructed to obtain an upsampled image; inputs the low-resolution image to be reconstructed into a preset shallow feature expression extraction module to obtain shallow features of the image; inputs the shallow features of the image into a preset multi-branch feature extraction module to obtain global features of the image; inputs the global features of the image and the shallow features of the image into a preset attention fusion network module to obtain image fusion features; the attention fusion network module includes fusing spatial attention with pixel attention; the image fusion features and the upsampled image are input into a preset upsampling network module to obtain a reconstructed super-resolution image. The present invention can effectively retain deep and shallow information, avoid the occurrence of shallow feature loss, enhance high-contribution feature information, suppress redundant information, and improve the reconstruction quality of super-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flowchart of a super-resolution image reconstruction method provided in Embodiment 1 of the present invention;

[0028] Figure 2 This is a schematic diagram of the structure of a neutron multi-branch feature extraction module according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of this embodiment can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Embodiment 1:

[0031] like Figure 1 As shown, an embodiment of the present invention provides a super-resolution image reconstruction method, comprising:

[0032] Step 1: Upsample the low-resolution image to be reconstructed to obtain an upsampled image;

[0033] Specifically, upsampling parameters are constructed according to the reconstruction magnification. For example, if the reconstruction magnification is 2, the upsampling step size is set to 2 to perform linear interpolation upsampling to obtain an upsampled image.

[0034] Step 2: inputting the low-resolution image to be reconstructed into a preset shallow feature expression extraction module to obtain shallow features of the image;

[0035] Specifically, the shallow feature expression extraction module is composed of a 3×3 convolution layer, and the extraction of shallow features is the result of the convolution operation between the image and the convolution layer.

[0036] Step 3: Input the shallow features of the image into a preset multi-branch feature extraction module to obtain the global features of the image;

[0037] Specifically, the multi-branch feature extraction module includes three sub-multi-branch feature extraction modules, each of which includes four links, wherein the features extracted by each link are fused with the features extracted by the adjacent links and then connected by a fully connected layer. In a specific embodiment, the process of the sub-multi-branch extraction module is as follows:

[0038]

[0039] ff1 is the result of shallow feature extraction. Its number of channels is one-fourth of the original input channels. The shallow feature ff1 is used as the input of the four links for feature extraction. MB ti (ff i ), i=1, 2, 3, 4 represents the second stage feature extraction, where the output result ff ti, i = 1, 2, 3, 4. The number of channels of the output result is consistent with ff1. Then, the feature output results of the second stage are combined in pairs and feature extraction is performed hierarchically. The feature extraction layer of the third stage is MB di , i = 1, 2, 3, add the output features of the second stage to the output features of the previous layer of the third stage, and then perform feature extraction. The output result ff di , i = 1, 2, 3, and the number of channels remains unchanged. This design can make full use of the information of the surface and deep layers while avoiding the loss of shallow feature information. Finally, all the extracted features are fused in the last stage and passed through a feature layer MB out (·) Extract features and output the final feature ff out .

[0040] In a specific embodiment, Figure 2 As shown in , except for the first link, each link combines the features of the current layer with the adjacent previous layer before extraction. This not only effectively retains the information of the features of the two adjacent layers, but also makes it easier to merge the adjacent features because of their small differences. It can be noted that the number of channels in the sub-multi-branch extraction module changes from 48 to 12. The number of channels in the middle layer has always been 12, and then the number of channels after combining the features becomes 48. The change from 48 to 12 makes the network width smaller, which will reduce the number of features in the middle layer and fundamentally reduce the number of module parameters. The changing number of channels is conducive to making full use of channel information. At the same time, the initial number of channels is the same as the final output number of channels, which ensures that the number of channels between the input features and the output features is consistent.

[0041] Step 4: inputting the global image features and the shallow image features into a preset attention fusion network module to obtain image fusion features; the attention fusion network module includes fusing spatial attention with pixel attention;

[0042] Specifically, the attention fusion network module includes three sub-attention fusion modules, each of which includes two branch networks, one of which includes a feature convolution layer, and the other includes an attention fusion residual module and a feature convolution layer. The attention fusion residual module includes a spatial attention mechanism fusion residual module and a pixel attention mechanism fusion residual module, and the spatial attention mechanism fusion residual module and the pixel attention mechanism fusion residual module are combined in parallel. In a specific embodiment, the algorithm flow of the attention fusion module is shown in the following formula:

[0043]

[0044] Two branches of CSPAR t1 (·) and CSPARt2 (·) represent the feature extraction layers of the two branches respectively. in After 1×1 feature extraction layer output s t1 With s t2 Features t1 After a 3×3 feature extraction layer CSPAR d1 (·) Generate feature s d1 Features t2 After feature extraction through the residual fusion module of the attention mechanism, the extracted features are subjected to feature extraction again CSPAR d2 (·), the output feature is s d2 The output features of the two branches are s d1 、s d2 The features of the two outputs are fused and then subjected to a final feature extraction) and the skip connection output s out .

[0045] Step 5: Input the image fusion features and the upsampled image into a preset upsampling network module to obtain a reconstructed super-resolution image.

[0046] Specifically, the upsampling network module adopts an inverse convolution operation, constructs an inverse convolution kernel according to the resolution of the upsampled image and the original low-resolution image to be reconstructed, and then performs inverse convolution on the extracted image fusion feature map using the convolution kernel to obtain a reconstructed super-resolution image.

[0047] The super-resolution image reconstruction method provided by the present invention can effectively retain deep and shallow information, avoid the loss of shallow features, enhance high-contribution feature information, suppress redundant information, and improve the reconstruction quality of super-resolution images.

[0048] Embodiment 2:

[0049] An embodiment of the present invention provides a super-resolution image reconstruction device, comprising:

[0050] Pre-upsampling module: upsamples the low-resolution image to be reconstructed to obtain an upsampled image;

[0051] Shallow feature extraction module: input the low-resolution image to be reconstructed into a preset shallow feature expression extraction module to obtain shallow features of the image;

[0052] Global feature extraction module: input the shallow features of the image into a preset multi-branch feature extraction module to obtain the global features of the image;

[0053] Feature fusion module: input the global image features and the shallow image features into a preset attention fusion network module to obtain image fusion features; the attention fusion network module includes fusing spatial attention with pixel attention;

[0054] Upsampling reconstruction module: input the image fusion features and the upsampled image into a preset upsampling module to obtain a reconstructed super-resolution image.

[0055] The super-resolution image reconstruction device provided by the present invention can effectively retain deep and shallow information, avoid the loss of shallow features, enhance high-contribution feature information, suppress redundant information, and improve the reconstruction quality of super-resolution images.

[0056] Embodiment three:

[0057] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a super-resolution image reconstruction method as described in the first embodiment is implemented.

[0058] Embodiment 4:

[0059] An embodiment of the present invention provides an electronic device, including:

[0060] A memory for storing instructions;

[0061] A processor is used to execute the instruction so that the device performs operations to implement the super-resolution image reconstruction method as described in any one of Embodiment 1.

[0062] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0063] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A super-resolution image reconstruction method, characterized in that: include: Upsampling the low-resolution image to be reconstructed to obtain an upsampled image; Inputting the low-resolution image to be reconstructed into a preset shallow feature expression extraction module to obtain shallow features of the image; Inputting the shallow features of the image into a preset multi-branch feature extraction module to obtain the global features of the image; Inputting the global image features and the shallow image features into a preset attention fusion network module to obtain image fusion features; the attention fusion network module includes fusing spatial attention with pixel attention; Inputting the image fusion feature and the upsampled image into a preset upsampled network module to obtain a reconstructed super-resolution image; The attention fusion network module includes three sub-attention fusion modules, each of which includes two branch networks, one of which includes a feature convolution layer, and the other includes an attention fusion residual module and a feature convolution layer; The attention fusion residual module includes a spatial attention mechanism fusion residual module and a pixel attention mechanism fusion residual module, and the spatial attention mechanism fusion residual module and the pixel attention mechanism fusion residual module are combined in parallel.

2. A super-resolution image reconstruction method according to claim 1, characterized in that: The multi-branch feature extraction module includes three sub-multi-branch feature extraction modules, each of which includes four links, wherein the features extracted by each link are fused with the features extracted by the adjacent links and then connected by a fully connected layer.

3. The super-resolution image reconstruction method according to claim 1, characterized in that: The spatial attention mechanism fuses the residual module, including an expanded convolution layer, a relu activation layer, a contracted convolution layer and a sigmoid activation layer, and the pixel attention mechanism fuses the residual module, including a 1×1 convolution layer and a sigmoid activation layer.

4. A super-resolution image reconstruction device, characterized in that: include: Pre-upsampling module: upsamples the low-resolution image to be reconstructed to obtain an upsampled image; Shallow feature extraction module: input the low-resolution image to be reconstructed into a preset shallow feature expression extraction module to obtain shallow features of the image; Global feature extraction module: input the shallow features of the image into a preset multi-branch feature extraction module to obtain the global features of the image; Feature fusion module: input the global image features and the shallow image features into a preset attention fusion network module to obtain image fusion features; the attention fusion network module includes fusing spatial attention with pixel attention; Upsampling reconstruction module: inputting the image fusion feature and the upsampled image into a preset upsampling module to obtain a reconstructed super-resolution image; The attention fusion network module includes three sub-attention fusion modules, each of which includes two branch networks, one of which includes a feature convolution layer, and the other includes an attention fusion residual module and a feature convolution layer; The attention fusion residual module includes a spatial attention mechanism fusion residual module and a pixel attention mechanism fusion residual module, and the spatial attention mechanism fusion residual module and the pixel attention mechanism fusion residual module are combined in parallel.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the super-resolution image reconstruction method as described in any one of claims 1 to 3 is implemented.

6. An electronic device, characterized in that: include: A memory for storing instructions; A processor is used to execute the instruction so that the device performs the operation of implementing the super-resolution image reconstruction method as described in any one of claims 1 to 3.

Images