Image super-resolution reconstruction method and device
Feature extraction and fusion of images through multi-scale cascaded attention residual networks, solving the problems of difficulty in deep network training and poor low-frequency information processing in the prior art, and achieving high-quality image super-resolution reconstruction.
Patent Information
- Application Number
- CN202210147765.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-02-17
AI Technical Summary
Existing image super-resolution reconstruction algorithms require very deep neural network architectures, which are long and difficult to train. They cannot fully utilize features on multiple scales, limiting learning ability, and low-frequency information in low-resolution inputs convolution Neural network learning has hindered.
Using image super-resolution reconstruction method based on multi-scale cascading attention residual network, feature extraction, fusion and enhancement are performed through shallow feature extraction, multi-scale cascading attention residual module and global jump connection, using sub-pixel volume The high-resolution image is finally reconstructed.
This achieves image reconstruction with richer details and higher quality, bypasses low-frequency information, integrates features at different levels of network depth, and avoids the establishment of a very deep network and still obtains good high-resolution reconstruction effects.
Smart Images

Figure CN114581300B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image super-resolution reconstruction, and in particular to an image super-resolution reconstruction method and device. Background Art
[0002] With the rapid development of computer technology and artificial intelligence, high-resolution words such as 8k and 100 million pixels continue to appear in people's vision, which shows that people's demand for high-resolution images is increasing. In addition, in the fields of security monitoring, medical imaging, remote sensing, face recognition, etc., images are important information carriers. High-quality images can provide richer details and obtain more usable information. Therefore, improving the resolution of images will be of great significance in real life.
[0003] In recent years, people have higher and higher requirements for image resolution. Super-resolution reconstruction, as a low-level visual task, has become one of the hot topics in the field of computer vision. Super-resolution solves the problem of reconstructing a high-resolution output image from a low-resolution input image. By continuously enriching the details of the image, the visual quality of the image is better and clearer.
[0004] With the continuous development of deep learning, deep neural networks have also been widely used in image super-resolution reconstruction, and have achieved good reconstruction results. However, the current mainstream algorithms often require very deep architectures and long training times, and the deeper the network, the more difficult it is to train, and the more training skills are required. At the same time, low-resolution input contains a wealth of low-frequency information, which is treated equally between channels, and will also hinder the learning of convolutional neural networks. In addition, the convolutional neural networks currently used for super-resolution cannot fully utilize features at multiple scales, which limits their learning ability. Therefore, it is very necessary to solve existing problems and reconstruct high quality. Summary of the invention
[0005] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the object of the present invention is to provide an image super-resolution reconstruction method and device based on a multi-scale cascaded attention residual network.
[0006] The technical solution adopted by the present invention is:
[0007] A method for super-resolution image reconstruction comprises the following steps:
[0008] Extract shallow features from low-resolution input images;
[0009] The shallow features are extracted, fused and enhanced through a backbone network consisting of m cascaded residual groups composed of multi-scale cascaded attention residual modules and global skip connections to obtain deep features.
[0010] Use sub-pixel convolution to upsample deep features;
[0011] The image is reconstructed using the features obtained from upsampling to obtain a higher resolution image.
[0012] Furthermore, the extraction of shallow features from the low-resolution input image includes:
[0013] Define a feature extraction component consisting of a convolutional layer to extract raw features from the low-resolution input image, as shown in the following formula:
[0014] F0=H SFE (I LR ) (1)
[0015] Among them, H SFE (·) represents the convolution operation applied to low-resolution feature extraction, I LR represents the low-resolution input image, and F0 represents the shallow features extracted by convolution.
[0016] Furthermore, the cascaded residual group includes n multi-scale cascaded attention residual modules, n feature splicing units, n feature compression units, n short jump connections and 1 local jump connection, and the feature compression unit consists of a 1x1 convolution.
[0017] Furthermore, the formula expression of the cascaded residual group is as follows:
[0018] F m,1 =H MCRAB (F m-1 ) (2)
[0019] F m,2 =H MCRAB (w 1×1 *[F m,1 , F m-1 ]+b) (3)
[0020] …
[0021] F m,n =H MCRAB (w 1×1 *[F m,n-1 , F m,n-2 ]+b) (4)
[0022] F m =F m,n +F m-1 (5)
[0023] Among them, F m,nH represents the output feature of the nth multi-scale cascaded attention residual module in the mth cascaded residual group, MCRAB represents the operation of the multi-scale cascaded attention residual module, F m represents the output features of the mth cascade residual group, [] represents feature concatenation, and w 1×1 represents the weight of the 1x1 convolution, and b represents the bias of the convolution kernel.
[0024] Furthermore, the multi-scale cascaded attention residual module includes an attention residual unit, an error feedback fusion unit, a skip connection and a cascade operation;
[0025] The formula expression of the multi-scale cascaded attention residual module is as follows:
[0026] F 3×3,in1 =w 1×1 *F m,n-1 +b (6)
[0027] F 3,1 =H RAB,3×3 (F 3×3,in1 )+F 3×3,in1 (7)
[0028] F 3×3,in2 =w 1×1 *[F 3,1 , F 3×3,in1 ]+b (8)
[0029] F 3,2 =H RAB,3×3 (F 3×3,in2 )+F 3×3,in2 (9)
[0030] F 3×3_in3 =w 1×1 *[F 3,2 , F 3,1 , F 3×3,in1 ]+b (10)
[0031] F 5×5,in1 =w 1×1 *F m,n-1 +b (11)
[0032] F 5,1 =H RAB,5×5 (F 5×5,in1 )+F 5×5,in1 (12)
[0033] F 5×5,in2 =w 1×1 *[F 5,1 , F 5×5,in1 ]+b (13)
[0034] F 5,2 =H RAB,5×5 (F 5×5,in2 )+F 5×5,in2 (14)
[0035] F 5×5,in3 =w 1×1 *[F 5,2 , F 5,1 , F 5×5,in1 ]+b (15)
[0036] F m,n =H Confusion (F 3×3,in3 , F 5×5,in3 )+F m,n-1 (16)
[0037] Among them, F 3×3,in1 、F 3×3,in2 、F 3×3,in3 They represent the input features at different stages when the scale is 3x3, F 3,1 、F 3,2 They represent the intermediate features when the scale is 3x3, F 5×5,in1 、F 5×5,in2 、F 5×5,in3 They represent the input features at different stages when the scale is 5x5, F 5,1 、F 5,2 They represent the intermediate features when the scale is 5x5, H RAB,3×3 , H RAB,5×5 They represent the attention residual units with convolution kernels of 3x3 and 5x5, respectively, and H Confusion represents the error feedback fusion unit, F m,n represents the output feature of the nth multi-scale cascaded attention residual module in the mth cascaded residual group, [] represents feature concatenation, and w 1×1 represents the weight of the 1x1 convolution, and b represents the bias of the convolution kernel.
[0038] Furthermore, the attention residual unit adopts a wide activation method to obtain wider channel features without changing the number of parameters. At the same time, the channel attention module is used to enhance the channel features before activation, and finally the spatial attention unit is used to enhance the spatial features of the residual. Both attention mechanisms adopt a combination of average pooling and maximum pooling.
[0039] The formula expression of the attention residual unit is as follows:
[0040] y=τ(H CA (w k×k *x+b)) (17)
[0041] Fr =H SA (w k×k *y+b) (18)
[0042] Among them, x and y represent the input features and output features respectively, τ represents the nonlinear activation function ReLU, and w k×k represents the weight of the kxk convolution, H CA represents the channel attention unit, H SA represents the channel attention unit, where the formula of each attention unit is as follows:
[0043] H CA =σ(H FC (τ(H FC (P Avg (x)+P Max (x)))))*x (19)
[0044] H SA =σ(w 7×7 *[(P Avg (x), P Max (x)]+b))*x (20)
[0045] Among them, σ represents the nonlinear activation function Sigmoid, w 7×7 is the weight of the 7x7 convolution, P Avg represents the average pooling operation, P Max represents the maximum pooling operation, H FC represents a fully connected layer.
[0046] Furthermore, the formula expression of the error feedback fusion unit is as follows:
[0047] f feedback_3 =τ(w 3×3 *F 3×3,in3 +b)-τ(w 3×3 *F 5×5,in3 +b) (21)
[0048] F 3×3 =τ(w 3×3 *f feedback_3 +b)+F 3×3,in3 (twenty two)
[0049] f feedback_5 =τ(w 3×3 *F 5×5,in3 +b)-τ(w 3×3 *F 3×3,in3 +b) (23)
[0050] F 5×5 =τ(w 3×3*f feedback_5 +b)+F 5×5,in3 (twenty four)
[0051] F confusion =w 3×3 *[F 3×3 , F 5×5 ]+b (25)
[0052] Among them, f feedback_3 、f feedback_5 Respectively represent the error feedback between different scales; F 3×3 , F 5×5 They represent the fusion features after error feedback, τ represents the nonlinear activation function ReLU, and w 3×3 is the weight of the 3x3 convolution, F confusion Represents multi-scale residual fusion features.
[0053] Furthermore, the shallow features are extracted, fused and enhanced through a backbone network consisting of m cascaded residual groups composed of multi-scale cascaded attention residual modules and global skip connections to obtain deep features, including:
[0054] The shallow features are passed through a backbone network consisting of m cascaded residual groups, a residual feature extraction backbone connected in series with a 3x3 convolution, and a global skip connection to obtain deep features; the specific formula is as follows:
[0055] F m =H crir,m (F m-1 )=H crir,m (H crir,m-1 (…(H crir,1 (F0)…)) (26)
[0056] F Res =τ(w 3×3 *F m +b) (27)
[0057] F DF =F0+F Res (28)
[0058] Among them, H crir,m is the operation of the mth cascade residual group, F Res is the final residual feature after passing through a 3x3 convolutional layer, F DF It is the deep feature finally composed of shallow features and residual features, and F0 is the shallow feature.
[0059] Furthermore, the upsampling of deep features using sub-pixel convolution includes:
[0060] A layer of sub-pixel convolution is used to upsample the deep features extracted by the backbone network. The specific formula is as follows:
[0061] F up =H Sub_pixel (F DF ) (29)
[0062] Among them, H Sub_pixel For the upsampling operation using sub-pixel convolution, F up is the upsampled output feature.
[0063] The method of reconstructing the image using the features obtained by upsampling to obtain an image with a higher resolution includes:
[0064] Reconstruct high-resolution image by upsampling predicted features I SR , the specific formula is as follows:
[0065] I SR =H R (F up ) (30)
[0066] Among them, H R To reconstruct a high-resolution image I SR The convolution operation.
[0067] Another technical solution adopted by the present invention is:
[0068] An image super-resolution reconstruction device, comprising:
[0069] at least one processor;
[0070] at least one memory for storing at least one program;
[0071] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0072] The beneficial effects of the present invention are as follows: the present invention adopts a multi-scale cascaded attention residual module, extracts, enhances and fuses multiple features from the perspectives of receptive field, width and attention; at the same time, through jump connections and cascaded residuals, bypasses low-frequency information, integrates features at different levels of network depth, and thus obtains richer details; through the method of the present invention, images with richer details and higher quality can be reconstructed. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0074] Figure 1 is a multi-scale cascaded attention residual network diagram in an embodiment of the present invention;
[0075] Figure 2 is an attention residual unit diagram in an embodiment of the present invention;
[0076] Figure 3 It is a diagram of a channel attention unit and a spatial attention unit in an embodiment of the present invention. DETAILED DESCRIPTION
[0077] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0078] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0079] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0080] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0081] like Figure 1 As shown, this embodiment provides an image super-resolution reconstruction method based on a multi-scale cascaded attention residual network graph. The extracted features are fused and enhanced through a multi-scale cascaded attention residual module, and the deep residual features are learned by using a global residual embedded cascade residual structure. Finally, the reconstruction is completed, which specifically includes the following steps:
[0082] Step 1: Perform shallow feature extraction on the input image, specifically:
[0083] A feature extraction component consisting of a 3x3 convolutional layer is used to extract the original features from the low-resolution input, as shown in the following formula:
[0084] F0=H SFE (I LR ) (1)
[0085] Among them, H SFE (·) represents the convolution operation applied to low-resolution feature extraction, I LR represents the low-resolution input image, and F0 represents the shallow features extracted by convolution.
[0086] Step 2: Attention residuals can extract features of different scales by using convolution kernels of different sizes, so that the network can learn richer image information. Figure 1 As shown in , by inputting shallow features into a backbone network composed of m cascaded residual groups and global skip connections, shallow features can be extracted and enhanced to obtain richer and deeper features. The cascaded residual group contains n multi-scale cascaded residual attention modules, such as Figure 1 As shown in FIG. 1 , each multi-scale residual attention module includes 4 attention residual units, 1 error feedback fusion unit and 6 short skip connections. In this example, m=3 and n=3 are taken, but the values of m and n do not constitute a limitation on the technical solution of the present invention. The attention residual units use 3x3 and 5x5 convolution kernels respectively. The attention residual units are as follows: Figure 2 As shown, the channel attention unit and the spatial attention unit involved are as follows Figure 3 As shown; the error feedback fusion unit is as follows Figure 1 As shown in the figure, the fusion module is composed of 4 3x3 convolutions followed by relu, feature concatenation units and 3x3 convolutions. The feature concatenation unit is composed of 1x1 convolutions. The use of residual learning makes the network easier to optimize. The specific process is as follows:
[0087] First, the shallow features extracted in step 1 are passed through the attention residual unit to obtain two different features, and then these two features are fused through the error feedback fusion unit, and finally added to the shallow features to form a residual block; the expression is as follows:
[0088] The specific formula is as follows:
[0089] F 3×3,in1 =w 1×1 *F m,n-1 +b (2)
[0090] F 3,1 =H RAB,3×3 (F 3×3,in1 )+F 3×3,in1 (3)
[0091] F 3×3,in2 =w 1×1 *[F 3,1 , F 3×3,in1 ]+b (4)
[0092] F 3,2 =H RAB,3×3 (F 3×3,in2 )+F 3×3,in2 (5)
[0093] F 3×3_in3 =w 1×1 *[F 3,2 , F 3,1 , F 3×3,in1 ]+b (6)
[0094] F 5×5,in1 =w 1×1 *F m,n-1 +b (7)
[0095] F 5,1 =H RAB,5×5 (F 5×5,in1 )+F 5×5,in1 (8)
[0096] F 5×5,in2 =w 1×1 *[F 5,1 , F 5×5,in1 ]+b (9)
[0097] F 5,2 =H RAB,5×5 (F 5×5,in2 )+F 5×5,in2 (10)
[0098] F 5×5,in3 =w 1×1 *[F 5,2 , F5,1 , F 5×5,in1 ]+b (11)
[0099] F m,n =H Confusion (F 3×3,in3 , F 5×5,in3 )+F m,n-1 (12)
[0100] Among them, F 3×3,in1 、F 3×3,in2 、F 3×3,in3 They represent the input features at different stages when the scale is 3x3, F 3,1 、F 3,2 They represent the intermediate features when the scale is 3x3, F 5×5,in1 、F 5×5in2 、F 5×5,in3 They represent the input features at different stages when the scale is 5x5, F 5,1 、F 5,2 They represent the intermediate features when the scale is 5x5, H RAB,3×3 , H RAB,5×5 They represent the attention residual units with convolution kernels of 3x3 and 5x5, respectively, and H Confusion represents the error feedback fusion unit. In this example, m is [1, 3], and n is [1, 3].
[0101] Step 3: The shallow features are extracted through a feature extraction network consisting of m cascaded residual groups and a 3x3 convolution in series residual feature extraction backbone and global skip connection, thereby obtaining rich features required for upsampling. The specific formula is as follows:
[0102] F m =H crir,m (F m-1 )=H crir,m (H crir,m-1 (…(H crir,1 (F0)…)) (13)
[0103] F Res =τ(w 3×3* F m +b) (14)
[0104] F DF =F0+F Res (15)
[0105] Among them, H crir,m is the operation of the mth cascade residual group, F Res is the final residual feature after passing through a 3x3 convolutional layer, F DF It is the final deep feature composed of shallow features and residual features.
[0106] Step 4: Use a sub-pixel convolution layer to upsample the deep features extracted by the backbone network to obtain a high-resolution image. The expression is as follows:
[0107] F up =H Sub_pixel (F DF ) (16)
[0108] Among them, H Sub_pixel For the upsampling operation using sub-pixel convolution, F up is the upsampled output feature.
[0109] Step 4: Reconstruct the image, specifically: reconstruct the high-resolution image I by upsampling the predicted features SR , the expression is as follows:
[0110] I SR =H R (F up ) (15)
[0111] Among them, H R To reconstruct a high-resolution image I SR The convolution operation.
[0112] In summary, the present invention has the following advantages and effects compared with the prior art:
[0113] (1) The embodiment of the present invention extracts multiple scale features from the image by adopting a multi-scale cascaded attention residual module. At the same time, under the premise of ensuring that the parameters remain unchanged, the channel features before the activation function are broadened, and the channel features and spatial features in the residual are enhanced by using channel attention and spatial attention respectively. Finally, through the error feedback fusion unit, the error between scales is used to enhance the scale features of each scale, and the multi-scale enhanced features are fused, so that the extracted features are richer.
[0114] (2) The embodiment of the present invention adopts a structure of embedding cascaded residuals in the global residual, so that the network can bypass the low-frequency information in the low-resolution input, learn more high-frequency residual information, and obtain rich detail features. In addition, it is not necessary to build a very deep network to obtain a high-resolution reconstructed image with good effect.
[0115] This embodiment also provides an image super-resolution reconstruction device, including:
[0116] at least one processor;
[0117] at least one memory for storing at least one program;
[0118] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0119] An image super-resolution reconstruction device of this embodiment can execute an image super-resolution reconstruction method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0120] This embodiment also provides a storage medium, which stores instructions or programs that can execute an image super-resolution reconstruction method provided by an embodiment of the method of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.
[0121] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0122] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0123] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0125] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0126] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0127] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0128] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0129] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A method for super-resolution image reconstruction, characterized in that: The following steps are involved: Extract shallow features from low-resolution input images; The shallow features are extracted, fused and enhanced through a backbone network consisting of m cascaded residual groups composed of multi-scale cascaded attention residual modules and global skip connections to obtain deep features. Use sub-pixel convolution to upsample deep features; Reconstruct the image using the features obtained by upsampling to obtain a higher resolution image; The cascade residual group includes n multi-scale cascaded attention residual modules, n feature concatenation units, n feature compression units, n short jump connections and 1 local jump connection, and the feature compression unit consists of a 1x1 convolution; The formula expression of the cascaded residual group is as follows: F m,1 =H MCRAB (F m-1 ) (2) F m,2 =H MCRAB (w 1×1 *[F m,1 ,F m-1 ]+b) (3) … F m,n =H MCRAB (w 1×1 *[F m,n-1 ,F m,n-2 ]+b) (4) F m =F m,n +F m-1 (5) Among them, F m,n represents the output features of the nth multi-scale cascaded attention residual module in the mth cascaded residual group, H MCRAB represents the operation of the multi-scale cascaded attention residual module, F m represents the output features of the mth cascade residual group, [] represents feature concatenation, and w 1×1 represents the weight of 1x1 convolution, and b represents the bias of the convolution kernel; The multi-scale cascaded attention residual module includes an attention residual unit, an error feedback fusion unit, a skip connection and a cascade operation; The formula expression of the multi-scale cascaded attention residual module is as follows: F 3×3,in1 =w 1×1 *F m,n-1 +b (6) F 3,1 =H RAB,3×3 (F 3×3,in1 )+F 3×3,in1 (7) F 3×3,in2 =w 1×1 *[F 3,1 ,F 3×3,in1 ]+b (8) F 3,2 =H RAB,3×3 (F 3×3,in2 )+F 3×3,in2 (9) F 3×3_in3 =w 1×1 *[F 3,2 ,F 3,1 ,F 3×3,in1 ]+b (10) F 5×5,in1 =w 1×1 *F m,n-1 +b (11) F 5,1 =H RAB,5×5 (F 5×5,in1 )+F 5×5,in1 (12) F 5×5,in2 =w 1×1 *[F 5,1 ,F 5×5,in1 ]+b (13) F 5,2 =H RAB,5×5 (F 5×5,in2 )+F 5×5,in2 (14) F 5×5,in3 =w 1×1 *[F 5,2 ,F 5,1 ,F 5×5,in1 ]+b (15) F m,n =H Confusion (F 3×3,in3 ,F 5×5,in3 )+F m,n-1 (16) Among them, F 3×3,in1 、F 3×3,in2 、F 3×3,in3 They represent the input features at different stages when the scale is 3x3, F 3,1 、F 3,2 They represent the intermediate features when the scale is 3x3, F 5×5,in1 、F 5×5,in2 、F 5×5,in3 They represent the input features at different stages when the scale is 5x5, F 5,1 、F 5,2 They represent the intermediate features when the scale is 5x5, H RAB,3×3 , H RAB,5×5 They represent the attention residual units with convolution kernels of 3x3 and 5x5, respectively, and H Confusion represents the error feedback fusion unit, F m,n represents the output feature of the nth multi-scale cascaded attention residual module in the mth cascaded residual group, [] represents feature concatenation, and w 1×1 represents the weight of the 1x1 convolution, and b represents the bias of the convolution kernel.
2. The image super-resolution reconstruction method according to claim 1, characterized in that: The extraction of shallow features from the low-resolution input image includes: Define a feature extraction component consisting of a convolutional layer to extract raw features from the low-resolution input image, as shown in the following formula: F0=H SFE (I LR ) (1) Among them, H SFE (·) represents the convolution operation applied to low-resolution feature extraction, I LR represents the low-resolution input image, and F0 represents the shallow features extracted by convolution.
3. The image super-resolution reconstruction method according to claim 1, characterized in that: The attention residual unit adopts a wide activation method to obtain wider channel features without changing the number of parameters. At the same time, the channel attention module is used to enhance the channel features before activation, and finally the spatial attention unit is used to enhance the spatial features of the residual. Both attention mechanisms adopt a combination of average pooling and maximum pooling. The formula expression of the attention residual unit is as follows: y=τ(H CA (w k×k *x+b)) (17) F r =H SA (w k×k *y+b) (18) Among them, x and y represent the input features and output features respectively, τ represents the nonlinear activation function ReLU, and w k×k represents the weight of the kxk convolution, H CA represents the channel attention unit, H SA represents the channel attention unit, where the formula of each attention unit is as follows: H CA =σ(H FC (τ(H FC (P Avg (x)+P Max (x)))))*x (19) H SA =σ(w 7×7 *[(P Avg (x),P Max (x)]+b))*x (20) Among them, σ represents the nonlinear activation function Sigmoid, w 7×7 is the weight of the 7x7 convolution, P Avg represents the average pooling operation, P Max represents the maximum pooling operation, H FC represents a fully connected layer.
4. The image super-resolution reconstruction method according to claim 1, characterized in that: The formula expression of the error feedback fusion unit is as follows: f feedback_3 =τ(w 3×3 *F 3×3,in3 +b)-τ(w 3×3 *F 5×5,in3 +b) (21) F 3×3 =τ(w 3×3 *f feedback_3 +b)+F 3×3,in3 (22) f feedback_5 =τ(w 3×3 *F 5×5,in3 +b)-τ(w 3×3 *F 3×3,in3 +b) (23) F 5×5 =τ(w 3×3 *f feedback_5 +b)+F 5×5,in3 (24) F confusion =w 3×3 *[F 3×3 ,F 5×5 ]+b (25) Among them, f feedback_3 、f feedback_5 Respectively represent the error feedback between different scales; F 3×3 ,F 5×5 They represent the fusion features after error feedback, τ represents the nonlinear activation function ReLU, and w 3×3 is the weight of the 3x3 convolution, F confusion Represents multi-scale residual fusion features.
5. The image super-resolution reconstruction method according to claim 1, characterized in that: The shallow features are extracted, fused and enhanced through a backbone network consisting of m cascaded residual groups composed of multi-scale cascaded attention residual modules and global skip connections to obtain deep features, including: The shallow features are passed through a backbone network consisting of m cascaded residual groups, a residual feature extraction backbone connected in series with a 3x3 convolution, and a global skip connection to obtain deep features; the specific formula is as follows: F m =H crir,m (F m-1 )=H crir,m (H crir,m-1 (…(H crir,1 (F0)…)) (26) F Res =τ(w 3×3 *F m +b) (27) F DF =F0+F Res (28) Among them, H crir,m is the operation of the mth cascade residual group, F Res is the final residual feature after passing through a 3x3 convolutional layer, F DF It is the deep feature finally composed of shallow features and residual features, and F0 is the shallow feature.
6. The image super-resolution reconstruction method according to claim 1, characterized in that: The sub-pixel convolution is used to upsample the deep features, including: A layer of sub-pixel convolution is used to upsample the deep features extracted by the backbone network. The specific formula is as follows: F up =H Sub_pixel (F DF ) (29) Among them, H Sub_pixel For the upsampling operation using sub-pixel convolution, F up is the upsampled output feature; The method of reconstructing the image using the features obtained by upsampling to obtain an image with a higher resolution includes: Reconstruct high-resolution image by upsampling predicted features I SR , the specific formula is as follows: I SR =H R (F up ) (30) Among them, H R To reconstruct a high-resolution image I SR The convolution operation.
7. An image super-resolution reconstruction device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.
