Image super-resolution method and system, computer equipment and storage medium

Abstract

The invention relates to the technical field of image super-resolution, and discloses an image super-resolution method and system, computer equipment and a storage medium, and the method comprises the steps: carrying out the sampling of a low-resolution image according to a sampling strategy based on region complexity, and obtaining a shallow feature; the shallow-layer features are input into a deep-layer feature extraction model to obtain deep-layer features, and the deep-layer feature extraction model is constructed by adopting a self-attention-based visual state space model; fusing the shallow-layer features and the deep-layer features to obtain image features; the image features are input into an image enhancement model for up-sampling and residual connection, a high-resolution image corresponding to the low-resolution image is obtained, and the image enhancement model is constructed by adopting a convolution-based pixel recombination method. According to the method, through multi-scale shallow feature extraction, global and local combined depth feature extraction and convolution residual error-based image reconstruction, the detail definition and the image resolution of the reconstructed image are effectively improved, and the method is suitable for various complex scenes.

Description

Technical Field

[0001] The present invention relates to the field of image super-resolution technology, and in particular to an image super-resolution method, system, computer equipment and storage medium. Background Art

[0002] With the widespread popularity of the Internet and mobile devices, the demand for image transmission and storage has increased dramatically. In order to save bandwidth and storage space, images usually need to be deeply compressed. However, deep compression often leads to loss of image details and blurriness, which seriously affects the visual experience and application effects. In order to compensate for the quality loss caused by compression, image super-resolution technology has emerged. Image super-resolution aims to restore high-resolution, clear images from low-resolution, blurred images, thereby improving the visual quality of images and making them appear more delicate and natural in various application scenarios.

[0003] At present, the commonly used image super-resolution methods include interpolation-based processing methods, mathematical model-based processing methods, and deep learning model-based processing methods. Although the existing image processing technology has made certain progress in many aspects, they still have the following significant limitations, which restrict their effectiveness and efficiency in practical applications. On the one hand, the repair effect is unstable. When the existing methods process different types of images and different degrees of damage, the repair effect is often unstable, and it is difficult to ensure consistent high-quality output. For example, the method based on the mathematical model is highly dependent on the prior knowledge of the image and is not effective when facing complex scenes. Although the classic deep learning model can learn the characteristics of the image, it still has the problem of insufficient generalization ability and is easily affected by the training data. On the other hand, the existing methods still have shortcomings in reconstructing the high-frequency details and texture of the image. For example, the interpolation-based method is difficult to effectively restore the edge and texture information of the image; the method based on the mathematical model is often too dependent on the preset model and it is difficult to capture the complex details of the image; although the classic deep learning model can learn some high-frequency information, it is difficult to completely restore the fine details of the image. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an image super-resolution method, system, computer device and storage medium, which can solve the problems of unstable restoration effect and insufficient detail reconstruction ability of existing methods, and achieve the technical effect of effectively improving the detail clarity and image resolution of the image.

[0005] In a first aspect, the present invention provides an image super-resolution method, the method comprising:

[0006] According to the sampling strategy based on regional complexity, the low-resolution image is sampled to obtain shallow features;

[0007] Inputting the shallow features into a pre-established deep feature extraction model to obtain deep features, wherein the deep feature extraction model is constructed using a visual state space model based on self-attention;

[0008] Fusing the shallow features with the deep features to obtain image features;

[0009] The image features are input into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image. The image enhancement model is constructed using a convolution-based pixel reconstruction method.

[0010] Furthermore, the step of sampling the low-resolution image according to the sampling strategy based on regional complexity to obtain shallow features includes:

[0011] Divide the low-resolution image into a number of image regions according to a preset size, and calculate the gradient of each pixel in each image region according to an edge detection algorithm;

[0012] Binarize the calculated gradient of each pixel based on the gradient threshold to obtain a binary gradient;

[0013] Calculate the proportion of pixels whose binary gradient is a preset gradient in each image region, and determine the complexity type of each image region according to a comparison result between the proportion and a proportion threshold, wherein the complexity type includes low complexity and high complexity;

[0014] According to the complexity type of each image area, a convolution kernel of a corresponding size is selected for sampling, and the obtained sampling results are spliced ​​and fused to obtain the shallow features of the low-resolution image.

[0015] Furthermore, the deep feature extraction model is composed of a plurality of feature processing modules and a convolutional layer connected in series, and the feature processing module is constructed using a visual state space model based on self-attention;

[0016] The feature processing module includes a convolution segmentation module, a feature extraction module and a splicing fusion module connected in series, and the feature extraction module includes a convolution residual module and a self-attention visual state space module connected in parallel;

[0017] The convolution residual module is composed of a plurality of convolution residual submodules connected in series, and the convolution residual submodule includes three convolution kernels of preset sizes and an activation function;

[0018] The self-attention visual state space module consists of a first residual submodule and a second residual submodule connected in series, wherein the first residual submodule includes a window multi-head self-attention submodule and a first normalization layer, and the second residual submodule includes a visual state space submodule and a second normalization layer.

[0019] Furthermore, the step of inputting the shallow features into a pre-established deep feature extraction model to obtain the deep features comprises:

[0020] Inputting the shallow features into the feature processing module for feature extraction, and inputting the extracted shallow extracted features into the next feature processing module, until the shallow extracted features output by the last feature processing module are obtained;

[0021] Inputting the shallow layer extracted features into the convolution layer for convolution operation to obtain deep layer features;

[0022] Wherein, the step of inputting the shallow features into the feature processing module for feature extraction includes:

[0023] Inputting the shallow features into the convolution segmentation module for average segmentation to obtain a first shallow feature and a second shallow feature;

[0024] Inputting the first shallow feature into the convolution residual module to obtain a local feature;

[0025] Inputting the second shallow feature into the self-attention visual state space module to obtain a global feature;

[0026] The local features and the global features are input into the splicing and fusion module to obtain shallow layer extraction features.

[0027] Furthermore, the image enhancement model includes a first convolutional layer, a plurality of pixel enhancement modules and a convolutional module, the pixel enhancement module and the convolutional module are connected in series, and the pixel enhancement module is connected in parallel with the first convolutional layer;

[0028] The pixel enhancement module includes a second convolution layer, and a third convolution layer, a pixel reorganization module and a fourth convolution layer connected in parallel with the second convolution layer, and the third convolution layer, the pixel reorganization module and the fourth convolution layer are connected in series.

[0029] Furthermore, the step of inputting the image features into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image includes:

[0030] Inputting the image features into the pixel enhancement module for upsampling, and inputting the obtained upsampled features into the next pixel enhancement module, until the upsampled features output by the last pixel enhancement module are obtained;

[0031] After the image features are input into the first convolutional layer for convolution operation, residual connection is performed with the up-sampled features, and the connected features are input into the convolution module for convolution operation to obtain a high-resolution image corresponding to the low-resolution image.

[0032] Furthermore, the step of inputting the image features into the pixel enhancement module for upsampling processing includes:

[0033] Upsampling the image features through the third convolution layer, the pixel reassembly module and the fourth convolution layer to obtain a first upsampling feature;

[0034] Performing a convolution operation on the image feature through the second convolution layer to obtain a first convolution feature;

[0035] Perform a residual connection on the first up-sampled feature and the first convolutional feature to obtain an up-sampled feature.

[0036] In a second aspect, the present invention provides an image super-resolution system, the system comprising:

[0037] A shallow feature extraction module is used to sample low-resolution images according to a sampling strategy based on regional complexity to obtain shallow features;

[0038] A deep feature extraction module, used for inputting the shallow features into a pre-established deep feature extraction model to obtain deep features, wherein the deep feature extraction model is constructed using a visual state space model based on self-attention;

[0039] A feature fusion module, used for fusing the shallow features with the deep features to obtain image features;

[0040] The image reconstruction module is used to input the image features into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image. The image enhancement model is constructed using a convolution-based pixel reconstruction method.

[0041] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0042] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0043] The present invention provides an image super-resolution method, system, computer device and storage medium. Through a multi-scale adaptive shallow feature extraction method based on dynamic convolution, shallow features can be extracted at multi-scale. Through a deep feature extraction model of a visual state space based on window multi-head self-attention, global features and local features are combined, which can significantly improve the accuracy and depth of feature extraction. At the same time, by combining local feature extraction with global context modeling and using an image enhancement model for high-quality reconstruction, the reconstructed image can present better details and overall effects. The present invention can effectively improve the detail clarity and image resolution of the reconstructed image, and is applicable to a variety of complex scenes, with good practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of a flow chart of an image super-resolution method in an embodiment of the present invention;

[0045] Figure 2 is another schematic flow chart of the image super-resolution method in an embodiment of the present invention;

[0046] Figure 3 yes Figure 2 Schematic diagram of the process of extracting mid- and shallow-level features;

[0047] Figure 4 is a schematic diagram of the structure of a deep extraction model in an embodiment of the present invention;

[0048] Figure 5 yes Figure 4 Schematic diagram of the structure of the convolution residual submodule;

[0049] Figure 6 yes Figure 4 Schematic diagram of the structure of the mid-window multi-head self-attention submodule;

[0050] Figure 7 yes Figure 4 Schematic diagram of the structure of the visual state space submodule;

[0051] Figure 8 is an iterative principle diagram of converting a 2D image into a 1D input sequence in an embodiment of the present invention;

[0052] Fig. 9 is a structural schematic diagram of an image enhancement model in an embodiment of the present invention;

[0053] Fig.10 is a schematic structural diagram of an image super-resolution system in an embodiment of the present invention;

[0054] Fig.11 It is a diagram of the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 , an image super-resolution method proposed in the first embodiment of the present invention, comprising steps S10 to S40:

[0057] Step S10, sampling the low-resolution image according to a sampling strategy based on regional complexity to obtain shallow features;

[0058] Step S20, inputting the shallow features into a pre-established deep feature extraction model to obtain deep features, wherein the deep feature extraction model is constructed using a visual state space model based on self-attention;

[0059] Step S30, fusing the shallow features and the deep features to obtain image features;

[0060] Step S40, inputting the image features into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image, wherein the image enhancement model is constructed using a convolution-based pixel reconstruction method.

[0061] In the present invention, the super-resolution processing steps of the image are as follows: Figure 2 As shown in the figure, it can be divided into three parts: shallow feature extraction, deep feature extraction and image reconstruction. Among them, the shallow feature extraction adopts a sampling strategy based on regional complexity. The specific steps include:

[0062] Divide the low-resolution image into a number of image regions according to a preset size, and calculate the gradient of each pixel in each image region according to an edge detection algorithm;

[0063] Binarize the calculated gradient of each pixel based on the gradient threshold to obtain a binary gradient;

[0064] Calculate the proportion of pixels whose binary gradient is a preset gradient in each image region, and determine the complexity type of each image region according to a comparison result between the proportion and a proportion threshold, wherein the complexity type includes low complexity and high complexity;

[0065] According to the complexity type of each image area, a convolution kernel of a corresponding size is selected for sampling, and the obtained sampling results are spliced ​​and fused to obtain the shallow features of the low-resolution image.

[0066] In this embodiment, a multi-scale feature extraction method with adaptive sampling is used to extract shallow features. Figure 3 First, the low-resolution image is segmented according to a certain size, such as 4×4, with a total of 16 image regions. Then, the Sobel edge detection algorithm is used to calculate the gradient of each pixel in the image, and the complexity of each image region is estimated based on the calculated gradient. Specifically, a horizontal Sobel filter is used for convolution to obtain a horizontal gradient map, and a vertical Sobel filter is used for convolution to obtain a vertical gradient map, and the gradient size of each pixel is obtained by taking the square root of the sum of squares; then, the gradient is binarized according to a preset threshold, and pixels with a gradient size greater than the threshold are marked as 1, and pixels with a gradient size less than the threshold are marked as 0. Finally, the proportion of pixels marked as 1 in the region is counted, and regions with a gradient of 1 greater than 50% are marked as high-complexity regions, and regions with a gradient of less than 50% are marked as low-complexity regions, thereby determining the complexity of each image region.

[0067] Finally, according to the complexity of the image area, different sampling strategies are adaptively selected for multi-scale sampling. Preferably, in high-complexity areas, smaller convolution kernels, such as 3×3 convolution kernels, are used for dense sampling to capture detail information; in low-complexity areas, larger convolution kernels, such as 5×5 convolution kernels, are used for sampling to reduce the impact of redundant information. After sampling at different scales, the sampling results are spliced ​​and fused to obtain shallow features of adaptive sampling.

[0068] In this embodiment, the feature extraction process is optimized through an adaptive multi-scale shallow feature extraction method, so that the extracted features focus more on the complex details of the image and reduce computational redundancy, which can more comprehensively capture the multi-scale information in the image and improve the feature extraction effect, thereby improving the overall performance of image super-resolution technology.

[0069] The extracted shallow features are input into a pre-established deep feature extraction model for feature extraction, thereby obtaining deep features. In this embodiment, the structure of the deep feature extraction model is as follows: Figure 4As shown, the deep feature extraction model is composed of several feature processing modules TMBR and a convolution layer Conv connected in series, wherein the feature processing module TMBR is constructed using a visual state space model based on self-attention. It should be noted that in this embodiment and subsequent embodiments, the convolution layer and the convolution operation are all represented by Conv.

[0070] Specifically, the feature processing module includes a convolutional segmentation module, a feature extraction module and a splicing and fusion module connected in series, wherein the convolutional segmentation module consists of a 1×1 convolutional layer Conv, an activation function LeakyReLu and a segmentation layer Split; the feature extraction module includes a convolutional residual module and a self-attention visual state space module connected in parallel; the convolutional residual module consists of a number of convolutional residual sub-modules RB connected in series, the self-attention visual state space module consists of a first residual sub-module and a second residual sub-module connected in series, the first residual sub-module includes a window multi-head self-attention sub-module WMSA and a first normalization layer LayerNom, the second residual sub-module includes a visual state space sub-module VSSM and a second normalization layer LayerNom; the splicing and fusion module consists of a splicing layer Concat and a convolutional fusion layer Conv connected in series.

[0071] Based on the above structure, the step of extracting shallow features through a deep feature extraction model in this embodiment includes:

[0072] Inputting the shallow features into the feature processing module for feature extraction, and inputting the extracted shallow extracted features into the next feature processing module, until the shallow extracted features output by the last feature processing module are obtained;

[0073] Inputting the shallow layer extracted features into the convolution layer for convolution operation to obtain deep layer features;

[0074] Wherein, the step of inputting the shallow features into the feature processing module for feature extraction includes:

[0075] Inputting the shallow features into the convolution segmentation module for average segmentation to obtain a first shallow feature and a second shallow feature;

[0076] Inputting the first shallow feature into the convolution residual module to obtain a local feature;

[0077] Inputting the second shallow feature into the self-attention visual state space module to obtain a global feature;

[0078] The local features and the global features are input into the splicing and fusion module to obtain shallow layer extraction features.

[0079] In this embodiment, the shallow feature is recorded as F, and the size is expressed as C×H F ×W F , where C represents the length, H F Indicates height, W F Represents the width. For F, it is first input to the convolutional segmentation module. The convolution kernel size of the convolution layer in this module is 1×1, and its output channel number is also C, with a size of C×H. F ×W F , and then split F into two features, denoted as F cnn and F tm , the size is C / 2×H F ×W F The advantage of doing this is that, on the one hand, the amount of model calculation can be effectively reduced by reducing the number of channels, and on the other hand, the two features can be processed independently and in parallel in the subsequent process, so that the features can be effectively extracted.

[0080] For the first shallow layer, extract the feature F cnn , a convolution operation is performed by a convolution residual module to extract local features. In this embodiment, the convolution residual module is composed of a plurality of convolution residual submodules RB connected in series, and preferably, three convolution residual submodules can be set. The structure of the convolution residual submodule is as follows: Figure 5 As shown, each RB consists of two 1×1 convolution kernels, a 3×3 convolution kernel and an activation function GELU. For the first shallow layer of the input, the feature F is extracted. cnn After the convolution operation, residual connection is performed to accelerate convergence. The feature obtained by the convolution residual module is the local feature F′ cnn .

[0081] For the second shallow layer, extract the feature F tm , then the self-attention visual state space module is input for processing. First, the multi-head attention calculation is performed through the first residual submodule. In this embodiment, the structure of the window multi-head self-attention submodule WMSA in the first residual submodule is as follows: Figure 6 As shown in the figure, Multi-Head Attention represents multi-head attention, ScaledDot-Product Attention represents scaled dot product attention, and the core idea of ​​WMSA is to divide the feature map into non-overlapping windows and perform multi-head self-attention calculations independently in each window. First, the input F tm By (C / 2×H F ×W F ) is reshaped into:

[0082]

[0083] Where M represents the window size.

[0084] Then the reshaped sequence is divided into multiple non-overlapping windows, each of which has a size of M. Thus, N windows of shape (M, M, C / 2) can be obtained. N can be expressed as:

[0085]

[0086] For each window W i , independently perform multi-head attention calculations. That is, after linearly projecting the three matrices QKV through the linear layer Liner, Q and K are inner-producted with MatMul, scaled and translated, masked, and normalized with SoftMax, and then inner-producted with V with MatMul, the output of each head is concatenated and the output of the window is obtained through a linear transformation Liner. Finally, the outputs of all windows are concatenated to obtain the final output sequence, that is, the output is reshaped into (C / 2, H F , W F ), and then the output of the WMSA and LayerNorm layers is residually connected to the initial input to obtain the output F t .

[0087] WMSA calculates the attention mask between different windows, but does not consider the relationship between windows. In order to model the pixel relationship between windows, the result F t The input is sent to the second residual submodule, and the long-range dependencies of the spatial dimension are modeled again through the visual state space submodule VSSM and the second normalization layer LayerNorm. VSSM converts the 2D image into a 1D input sequence by scanning in four directions and uses a discretized iterative formula for modeling, thereby efficiently capturing global information.

[0088] In this embodiment, the structure of VSSM is as follows: Figure 7 As shown, for the input F t , reshape it into (H F , W F , C / 2), denoted as F m , F m Will be sent to two parallel branches for processing:

[0089] Branch 1:

[0090] Feature F mEnter the linear layer Liner, expand the number of feature channels to λC. The expanded features enter the depthwise separable convolution layer DWConv. The output of DWConv enters the SiLU activation function, and then enters the 2D state space module 2DSSM, which scans the 2D feature map along four different directions (top left to bottom right, bottom right to top left, top right to bottom left, bottom left to top right), flattens the features into a 1D sequence, and converts the 2D image into a 1D input sequence for iteration. Figure 8 As shown. For each 1D sequence, a discrete state space equation is used to capture long-range dependencies. The sequences obtained by scanning in four directions are summed and merged. The merged sequence is reshaped into a 2D feature map. Finally, the normalization layer is input to normalize the features, and the output is recorded as F m1 .

[0091] Branch 2:

[0092] Feature F m Enter the linear layer Liner and expand the number of feature channels to λC. The expanded features enter the SiLU activation function and the output is recorded as F m2 .

[0093] F m1 and F m2 Perform element-wise multiplication to fuse the information of the two branches. The fused features enter the linear layer and project the number of channels back to C / 2. The final output is F′ m Shape and input features F m same.

[0094] The mathematical formula of the above process is expressed as:

[0095] F m1 =LN(2D-SSM(SiLU(DWConv(Linear(F m )))))

[0096] F m2 =SiLU(Linear(F m ))

[0097] F′ m =Linear(F m1 ⊙F m2 )

[0098] For the F′ output of VSSM m , and then normalized by the second normalization layer LayerNorm, and its output is the global feature F′ tm .

[0099] Finally, F′ tm and F′ cnnConcatenate Concat and fuse the results through a 1×1 convolution kernel Conv to obtain a low-resolution feature map containing rich information, that is, shallow extraction features.

[0100] The above steps are the feature extraction steps of a feature processing module. In this embodiment, the deep feature extraction model includes multiple feature processing modules connected in series. After being processed by multiple feature processing modules, a convolution operation is performed to generate a new deeper feature representation. That is, through the processing of multiple feature processing modules and convolution layers, shallow features are converted into a comprehensive description of the deep features of the original image, thereby obtaining deep features.

[0101] In this embodiment, the window multi-head self-attention and the visual state space model are combined to enable the deep feature extraction model to have the ability to capture pixel dependencies efficiently. The window multi-head self-attention reduces the computational complexity and improves the computational efficiency by performing self-attention operations within a local window. The combination of multiple convolution residuals effectively alleviates the problem of feature loss that may occur in the deep feature extraction process. The structural design of the deep feature extraction model in this embodiment not only improves the global context modeling capability of the model, but also ensures the accuracy and stability in complex image processing tasks.

[0102] After the shallow features and deep features are extracted through the above steps, the shallow features and deep features are fused, and the obtained image features are input into the image enhancement model for stepwise upsampling to reconstruct a high-quality image. In this embodiment, the structure of the image enhancement model is as follows: Fig. 9 As shown, the image enhancement model includes a first convolutional layer, several pixel enhancement modules PEB and a convolutional module, wherein each pixel enhancement module PEB is connected in series with the convolutional module, and the series-connected pixel enhancement modules PEB are connected in parallel with the first convolutional layer. The pixel enhancement module includes a second convolutional layer, and a third convolutional layer connected in parallel with the second convolutional layer, a pixel reorganization module pixel shuffle and a fourth convolutional layer, and the third convolutional layer, the pixel reorganization module pixel shuffle and the fourth convolutional layer are connected in series.

[0103] Based on the above structure, in this embodiment, the specific steps of performing image reconstruction on the input image features include:

[0104] Inputting the image features into the pixel enhancement module for upsampling, and inputting the obtained upsampled features into the next pixel enhancement module, until the upsampled features output by the last pixel enhancement module are obtained;

[0105] After the image features are input into the first convolutional layer for convolution operation, residual connection is performed with the up-sampled features, and the connected features are input into the convolution module for convolution operation to obtain a high-resolution image corresponding to the low-resolution image.

[0106] In this embodiment, three pixel enhancement modules PEB are preferably set and connected in series. For the input image feature Feature, after upsampling by three PEBs, residual connection is performed with the original input image feature, wherein the specific steps of upsampling by each PEB include:

[0107] Upsampling the image features through the third convolution layer, the pixel reassembly module and the fourth convolution layer to obtain a first upsampling feature;

[0108] Performing a convolution operation on the image feature through the second convolution layer to obtain a first convolution feature;

[0109] Perform a residual connection on the first up-sampled feature and the first convolutional feature to obtain an up-sampled feature.

[0110] In this embodiment, the PEB module is composed of multiple convolutional layers and a pixel reorganization module pixel shuffle. After the input image features are subjected to convolution, pixel shuffle upsampling and convolution in the PEB module, residual connection is performed with the input image features to obtain a complete upsampling operation of the PEB module. After the image features are subjected to upsampling processing of three serially connected PEB modules, residual connection is performed with the input image features to obtain complete upsampling features, wherein the purpose of the convolution operation in the residual connection is to ensure that the input and output image sizes are consistent.

[0111] The core principle of pixel shuffle is to rearrange the channel dimension of the feature map to the spatial dimension to achieve resolution improvement. For the input low-resolution feature map, assuming its size is H×W×C, where H is the height, W is the width, and C is the number of channels, the input feature channel C is first expanded, and the number of channels after expansion satisfies C'=C*r 2 , and then rearrange this feature map into a high-quality image with a size of r*H×r*W×C.

[0112] For input features Figure X And the output feature map Y, the pixel shuffle operation can be expressed as:

[0113]

[0114] Among them, i and j are the spatial indices of the output feature map, c is the channel index, represents rounding down, mod represents the modulo operation, and r is the upsampling factor.

[0115] After the upsampling is completed through the residual connection, the upsampling result is convolved by the convolution module. In this embodiment, the convolution module is a convolution layer with two 3×3 convolution kernels, that is, two consecutive convolution operations are performed using the 3×3 convolution kernel, so as to improve the details of the image layer by layer.

[0116] The image enhancement model of this embodiment effectively solves the problem of detail loss in the image reconstruction process by efficiently upsampling the input image features, designing multi-level convolutions, using residual connections, and performing end-to-end processing, thereby ensuring that the reconstructed and enhanced high-quality images have higher resolution, clearer details, and less noise.

[0117] The image super-resolution method provided in this embodiment has the problems of unstable restoration effect and insufficient detail reconstruction ability compared with the traditional method. The present invention can extract shallow features at multiple scales through a multi-scale adaptive shallow feature extraction method based on dynamic convolution, and combines global features with local features through a deep feature extraction model of the visual state space based on window multi-head self-attention, which significantly improves the accuracy and depth of feature extraction. At the same time, by combining local feature extraction with global context modeling and using the image reconstruction enhancement module for high-quality reconstruction, the reconstructed image can present better details and overall effect. The present invention effectively improves the detail clarity and image resolution of the reconstructed image, and is applicable to a variety of complex scenes, with good practicality.

[0118] See also Fig.10 Based on the same inventive concept, a second embodiment of the present invention provides an image super-resolution system, comprising:

[0119] A shallow feature extraction module 10 is used to sample the low-resolution image according to a sampling strategy based on regional complexity to obtain shallow features;

[0120] A deep feature extraction module 20, used for inputting the shallow features into a pre-established deep feature extraction model to obtain deep features, wherein the deep feature extraction model is constructed using a visual state space model based on self-attention;

[0121] A feature fusion module 30, used for fusing the shallow features with the deep features to obtain image features;

[0122] The image reconstruction module 40 is used to input the image features into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image. The image enhancement model is constructed using a convolution-based pixel reconstruction method.

[0123] The technical features and technical effects of the image super-resolution system proposed in the embodiment of the present invention are the same as those of the method proposed in the embodiment of the present invention, and are not described in detail here. Each module in the above-mentioned image super-resolution system can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0124] In addition, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0125] See also Fig.11 , an internal structure diagram of a computer device in an embodiment, the computer device can specifically be a terminal or a server. The computer device includes a processor, a memory, a network interface, a display and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image super-resolution method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0126] It can be understood by those skilled in the art that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have the same component arrangement.

[0127] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0128] In summary, an image super-resolution method, system, computer device and storage medium proposed in an embodiment of the present invention, the method samples a low-resolution image according to a sampling strategy based on regional complexity to obtain shallow features; inputs the shallow features into a pre-established deep feature extraction model to obtain deep features, the deep feature extraction model is constructed using a visual state space model based on self-attention; the shallow features and the deep features are fused to obtain image features; the image features are input into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image, the image enhancement model is constructed using a convolution-based pixel reconstruction method. The present invention can extract shallow features at multiple scales through a multi-scale adaptive shallow feature extraction method based on dynamic convolution, and combines global features with local features through a deep feature extraction model of a visual state space based on window multi-head self-attention, significantly improving the accuracy and depth of feature extraction, and at the same time, by combining local feature extraction with global context modeling, and using an image reconstruction enhancement module for high-quality reconstruction, the reconstructed image can present better details and overall effects. The present invention effectively improves the detail clarity and image resolution of the reconstructed image, is applicable to a variety of complex scenes, and has good practicability.

[0129] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. An image super-resolution method, characterized in that: include: According to the sampling strategy based on regional complexity, the low-resolution image is sampled to obtain shallow features; Inputting the shallow features into a pre-established deep feature extraction model to obtain deep features, wherein the deep feature extraction model is constructed using a visual state space model based on self-attention; Fusing the shallow features with the deep features to obtain image features; The image features are input into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image. The image enhancement model is constructed using a convolution-based pixel reconstruction method.

2. The image super-resolution method according to claim 1, characterized in that: The step of sampling the low-resolution image according to the sampling strategy based on regional complexity to obtain shallow features includes: Divide the low-resolution image into a number of image regions according to a preset size, and calculate the gradient of each pixel in each image region according to an edge detection algorithm; Binarize the calculated gradient of each pixel based on the gradient threshold to obtain a binary gradient; Calculate the proportion of pixels whose binary gradient is a preset gradient in each image region, and determine the complexity type of each image region according to a comparison result between the proportion and a proportion threshold, wherein the complexity type includes low complexity and high complexity; According to the complexity type of each image area, a convolution kernel of a corresponding size is selected for sampling, and the obtained sampling results are spliced ​​and fused to obtain the shallow features of the low-resolution image.

3. The image super-resolution method according to claim 1, characterized in that: The deep feature extraction model is composed of several feature processing modules and a convolution layer in series, and the feature processing module is constructed using a visual state space model based on self-attention; The feature processing module includes a convolution segmentation module, a feature extraction module and a splicing fusion module connected in series, and the feature extraction module includes a convolution residual module and a self-attention visual state space module connected in parallel; The convolution residual module is composed of a plurality of convolution residual submodules connected in series, and the convolution residual submodule includes three convolution kernels of preset sizes and an activation function; The self-attention visual state space module consists of a first residual submodule and a second residual submodule connected in series, wherein the first residual submodule includes a window multi-head self-attention submodule and a first normalization layer, and the second residual submodule includes a visual state space submodule and a second normalization layer.

4. The image super-resolution method according to claim 3, characterized in that: The step of inputting the shallow features into a pre-established deep feature extraction model to obtain the deep features comprises: Inputting the shallow features into the feature processing module for feature extraction, and inputting the extracted shallow extracted features into the next feature processing module, until the shallow extracted features output by the last feature processing module are obtained; Inputting the shallow layer extracted features into the convolution layer for convolution operation to obtain deep layer features; Wherein, the step of inputting the shallow features into the feature processing module for feature extraction includes: Inputting the shallow features into the convolution segmentation module for average segmentation to obtain a first shallow feature and a second shallow feature; Inputting the first shallow feature into the convolution residual module to obtain a local feature; Inputting the second shallow feature into the self-attention visual state space module to obtain a global feature; The local features and the global features are input into the splicing and fusion module to obtain shallow layer extraction features.

5. The image super-resolution method according to claim 1, characterized in that: The image enhancement model includes a first convolutional layer, a plurality of pixel enhancement modules and a convolutional module, the pixel enhancement module and the convolutional module are connected in series, and the pixel enhancement module is connected in parallel with the first convolutional layer; The pixel enhancement module includes a second convolution layer, and a third convolution layer, a pixel reorganization module and a fourth convolution layer connected in parallel with the second convolution layer, and the third convolution layer, the pixel reorganization module and the fourth convolution layer are connected in series.

6. The image super-resolution method according to claim 5, characterized in that: The step of inputting the image features into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image comprises: Inputting the image features into the pixel enhancement module for upsampling, and inputting the obtained upsampled features into the next pixel enhancement module, until the upsampled features output by the last pixel enhancement module are obtained; After the image features are input into the first convolutional layer for convolution operation, residual connection is performed with the up-sampled features, and the connected features are input into the convolution module for convolution operation to obtain a high-resolution image corresponding to the low-resolution image.

7. The image super-resolution method according to claim 6, characterized in that: The step of inputting the image features into the pixel enhancement module for upsampling processing comprises: Upsampling the image features through the third convolution layer, the pixel reassembly module and the fourth convolution layer to obtain a first upsampling feature; Performing a convolution operation on the image feature through the second convolution layer to obtain a first convolution feature; Perform a residual connection on the first up-sampled feature and the first convolutional feature to obtain an up-sampled feature.

8. An image super-resolution system, characterized in that: include: A shallow feature extraction module is used to sample low-resolution images according to a sampling strategy based on regional complexity to obtain shallow features; A deep feature extraction module, used for inputting the shallow features into a pre-established deep feature extraction model to obtain deep features, wherein the deep feature extraction model is constructed using a visual state space model based on self-attention; A feature fusion module, used for fusing the shallow features with the deep features to obtain image features; The image reconstruction module is used to input the image features into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image. The image enhancement model is constructed using a convolution-based pixel reconstruction method.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images