An image super-resolution method, system, computer device and storage medium

Abstract

The application relates to the technical field of image super-resolution, and discloses an image super-resolution method, system, computer device and storage medium, which comprises the following steps: according to a sampling strategy based on the complexity of a region, a low-resolution image is sampled to obtain shallow features; the shallow features are input into a deep feature extraction model to obtain deep features, and the deep feature extraction model is constructed by using a visual state space model based on self-attention; the shallow features and the deep features are fused to obtain image features; the image features are input into an image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image, and the image enhancement model is constructed by using a pixel recombination method based on convolution. Through multi-scale shallow feature extraction, global and local combined deep feature extraction and image reconstruction based on convolution and residual, the application effectively improves the detail definition and image resolution of the reconstructed image, and is suitable for various complex scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image super-resolution, and in particular to an image super-resolution method, system, computer device and storage medium. BACKGROUND

[0002] With the wide popularity of the Internet and mobile devices, the demand for image transmission and storage has increased dramatically. In order to save bandwidth and storage space, images often need to be deeply compressed. However, deep compression often leads to loss of image details and blurring, which seriously affects the visual experience and application effect. In order to make up for the quality loss caused by compression, image super-resolution technology has emerged. Image super-resolution aims to restore high-resolution, clear images from low-resolution, blurred images, thereby improving the visual quality of images and making them appear more delicate and natural in various application scenarios.

[0003] The commonly used image super-resolution methods at present include interpolation-based processing methods, mathematical model-based processing methods and deep learning model-based processing methods, etc. Although existing image processing technologies have made certain progress in many aspects, they still have the following significant limitations, which restrict their effectiveness and efficiency in practical applications. On the one hand, the repair effect is unstable. The repair effect of existing methods is often unstable when processing different types of images and different degrees of damage, and it is difficult to ensure consistent high-quality output. For example, the mathematical model-based method is highly dependent on the prior knowledge of the image, and the effect is not good when facing complex scenes. Although the classic deep learning model can learn the features of the image, it still has the problem of insufficient generalization ability and is easily affected by training data. On the other hand, the existing methods still have deficiencies in reconstructing the high-frequency details and textures of the image. For example, the interpolation-based method is difficult to effectively recover the edge and texture information of the image; the mathematical model-based method often relies too much on the preset model and is difficult to capture the complex details of the image; the classic deep learning model can learn some high-frequency information, but it is difficult to completely restore the fine details of the image. SUMMARY

[0004] In order to solve the above technical problems, the present application provides an image super-resolution method, system, computer device and storage medium, which can solve the problems of unstable repair effect and insufficient detail reconstruction capability of existing methods, and achieve the technical effect of effectively improving the detail clarity and image resolution of the image.

[0005] In a first aspect, the present application provides an image super-resolution method, which comprises:

[0006] According to the sampling strategy based on the complexity of the region, the low-resolution image is sampled to obtain shallow features;

[0007] input the shallow feature into a pre-established deep feature extraction model to obtain a deep feature, the deep feature extraction model being constructed based on a self-attention visual state space model;

[0008] fuse the shallow feature and the deep feature to obtain an image feature;

[0009] input the image feature into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image, the image enhancement model being constructed based on a convolution-based pixel reorganization method.

[0010] Further, the step of sampling the low-resolution image according to the sampling strategy based on the region complexity to obtain the shallow feature comprises:

[0011] divide the low-resolution image into a plurality of image regions according to a preset size, and calculate the gradient of each pixel point in each image region according to an edge detection algorithm;

[0012] binaryzation the calculated gradient of each pixel point based on a gradient threshold to obtain a binaryzation gradient;

[0013] calculate the proportion of pixel points with the binaryzation gradient being a preset gradient in each image region, and determine the complexity type of each image region according to the comparison result of the proportion and a proportion threshold, the complexity type including low complexity and high complexity;

[0014] select a convolution kernel of a corresponding size according to the complexity type of each image region to sample, and splice and fuse the obtained sampling result to obtain the shallow feature of the low-resolution image.

[0015] Further, the deep feature extraction model is composed of a plurality of feature processing modules and a convolution layer in series, the feature processing module being constructed based on a self-attention visual state space model;

[0016] the feature processing module comprises a convolution division module, a feature extraction module and a splicing and fusion module in series, the feature extraction module comprising a convolution residual module and a self-attention visual state space module in parallel;

[0017] the convolution residual module is composed of a plurality of convolution residual sub-modules in series, the convolution residual sub-module comprising three convolution kernels of a preset size and an activation function;

[0018] The self-attention visual state space module is composed of a first residual sub-module and a second residual sub-module connected in series, the first residual sub-module includes a window multi-head self-attention sub-module and a first normalization layer, and the second residual sub-module includes a visual state space sub-module and a second normalization layer.

[0019] Further, the step of inputting the shallow feature into a pre-established deep feature extraction model to obtain a deep feature comprises:

[0020] The shallow feature is input into the feature processing module for feature extraction, and the extracted shallow feature is input into the next feature processing module until the shallow feature output by the last feature processing module is obtained;

[0021] The shallow feature is input into the convolution layer for convolution operation to obtain a deep feature;

[0022] The step of inputting the shallow feature into the feature processing module for feature extraction comprises:

[0023] The shallow feature is input into the convolution segmentation module for average segmentation to obtain a first shallow feature and a second shallow feature;

[0024] The first shallow feature is input into the convolution residual module to obtain a local feature;

[0025] The second shallow feature is input into the self-attention visual state space module to obtain a global feature;

[0026] The local feature and the global feature are input into the splicing and fusion module to obtain a shallow extraction feature.

[0027] Further, the image enhancement model comprises a first convolution layer, a plurality of pixel enhancement modules and a convolution module, the pixel enhancement modules and the convolution module are connected in series, and the pixel enhancement modules are connected in parallel with the first convolution layer;

[0028] The pixel enhancement module comprises a second convolution layer, a third convolution layer, a pixel reorganization module and a fourth convolution layer connected in parallel with the second convolution layer, and the third convolution layer, the pixel reorganization module and the fourth convolution layer are connected in series.

[0029] Further, the step of inputting the image feature into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image comprises:

[0030] input the image feature into the pixel enhancement module for upsampling processing, and input the obtained upsampling feature into a next pixel enhancement module until the upsampling feature output by the last pixel enhancement module is obtained.

[0031] input the image feature into the first convolution layer for convolution operation, perform residual connection with the upsampling feature, input the connected feature into the convolution module for convolution operation, and obtain the high-resolution image corresponding to the low-resolution image.

[0032] Further, the step of inputting the image feature into the pixel enhancement module for upsampling processing comprises:

[0033] the image feature is subjected to upsampling processing through the third convolution layer, the pixel recombination module and the fourth convolution layer to obtain first upsampling feature;

[0034] the image feature is subjected to convolution operation through the second convolution layer to obtain first convolution feature;

[0035] the first upsampling feature and the first convolution feature are subjected to residual connection to obtain upsampling feature.

[0036] In a second aspect, the present application provides an image super-resolution system, which comprises:

[0037] a shallow feature extraction module configured to sample a low-resolution image according to a sampling strategy based on region complexity to obtain shallow features;

[0038] a deep feature extraction module configured to input the shallow features into a pre-established deep feature extraction model to obtain deep features, wherein the deep feature extraction model is constructed by using a self-attention-based visual state space model;

[0039] a feature fusion module configured to fuse the shallow features and the deep features to obtain image features;

[0040] an image reconstruction module configured to input the image features into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image, wherein the image enhancement model is constructed by using a convolution-based pixel recombination method.

[0041] In a third aspect, the present application further provides a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0042] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0043] The present application provides an image super-resolution method, system, computer device and storage medium. Through the multi-scale adaptive shallow feature extraction method based on dynamic convolution, the shallow features can be extracted in multiple scales. Through the deep feature extraction model of visual state space based on window multi-head self-attention, the global features and local features are combined, which can significantly improve the accuracy and depth of feature extraction. At the same time, by combining local feature extraction and global context modeling, and using an image enhancement model for high-quality reconstruction, the reconstructed image can present better details and overall effect. The present application can effectively improve the detail clarity and image resolution of the reconstructed image, and is suitable for various complex scenes, and has good practicability. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of the image super-resolution method in the embodiments of the present application;

[0045] Figure 2 is another flowchart of the image super-resolution method in the embodiments of the present application;

[0046] Figure 3 is Figure 2 a flowchart of shallow feature extraction;

[0047] Figure 4 is a structural diagram of the deep extraction model in the embodiments of the present application;

[0048] Figure 5 is Figure 4 a structural diagram of the convolution residual submodule;

[0049] Figure 6 is Figure 4 a structural diagram of the window multi-head self-attention submodule;

[0050] Figure 7 is Figure 4 a structural diagram of the visual state space submodule;

[0051] Figure 8 is an iterative schematic diagram of converting a 2D image into a 1D input sequence in the embodiments of the present application;

[0052] Figure 9 is a structural diagram of the image enhancement model in the embodiments of the present application;

[0053] Figure 10 is a structural diagram of the image super-resolution system in the embodiments of the present application;

[0054] Figure 11 is the internal structure diagram of the computer device in the embodiment of the application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0056] Referring to Figure 1 , the first embodiment of the present application provides an image super-resolution method, which comprises steps S10-S40.

[0057] In step S10, a low-resolution image is sampled according to a sampling strategy based on region complexity to obtain shallow features.

[0058] In step S20, the shallow features are input into a pre-established deep feature extraction model to obtain deep features, and the deep feature extraction model is constructed by using a visual state space model based on self-attention.

[0059] In step S30, the shallow features and the deep features are fused to obtain image features.

[0060] In step S40, the image features are input into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image, and the image enhancement model is constructed by using a pixel recombination method based on convolution.

[0061] In the present application, the super-resolution processing steps of the image can be divided into three parts of shallow feature extraction, deep feature extraction and image reconstruction as shown in Figure 2 , wherein the shallow feature extraction adopts a sampling strategy based on region complexity, and the specific steps include:

[0062] The low-resolution image is divided into a plurality of image regions according to a preset size, and the gradient of each pixel point in each image region is calculated according to an edge detection algorithm.

[0063] Each pixel point gradient calculated is binarized based on a gradient threshold to obtain a binarized gradient.

[0064] The proportion of pixels with binary gradient in each image region is calculated, and the complexity type of each image region is determined according to the comparison result of the proportion and the proportion threshold, wherein the complexity type includes low complexity and high complexity.

[0065] According to the complexity type of each image region, a corresponding size of convolution kernel is selected for sampling, and the obtained sampling results are spliced and fused to obtain the shallow features of the low-resolution image.

[0066] In this embodiment, a multi-scale feature extraction method with adaptive sampling is used for shallow feature extraction. Please refer to Figure 3 , first, the low-resolution image is segmented into a certain size, such as 4x4, a total of 16 image regions, then the Sobel edge detection algorithm is used to calculate the gradient of each pixel in the image, and the complexity of each region of the image is estimated according to the calculated gradient. Specifically, a horizontal Sobel filter is used for convolution operation to obtain a horizontal gradient map, and a vertical Sobel filter is used for convolution operation to obtain a vertical gradient map, and the gradient size of each pixel is obtained by squaring and square root; then the gradient is binarized according to the pre-set threshold, the pixel points with gradient size greater than the threshold are marked as 1, and the pixel points with gradient size less than the threshold are marked as 0, and finally the proportion of pixel points marked as 1 in the region is counted, the region with gradient of 1 and proportion of more than 50% is marked as high complexity region, and the region with gradient of 1 and proportion of less than 50% is marked as low complexity region, so as to determine the complexity of each image region.

[0067] Finally, through the complexity of the image region, different sampling strategies are adaptively selected for multi-scale sampling. Preferably, in the high complexity region, a smaller convolution kernel, such as a 3x3 size convolution kernel, is used for dense sampling to capture detailed information; in the low complexity region, a larger convolution kernel, such as a 5x5 size convolution kernel, is used for sampling to reduce the influence of redundant information. After sampling of different scales, the sampling results are spliced and fused to obtain the adaptive sampling shallow features.

[0068] In this embodiment, the feature extraction process is optimized by the adaptive multi-scale shallow feature extraction method, so that the extracted features pay more attention to the complex details of the image, and the calculation redundancy is reduced, the multi-scale information in the image can be captured more comprehensively, the feature extraction effect is improved, and the overall performance of the image super-resolution technology is improved.

[0069] The extracted shallow features are input into a pre-established deep feature extraction model for feature extraction, so as to obtain deep features. In this embodiment, the structure of the deep feature extraction model is as follows Figure 4As shown, the deep feature extraction model is composed of a plurality of feature processing modules TMBR and a convolution layer Conv in series, wherein the feature processing module TMBR is constructed based on a self-attention visual state space model. It should be noted that in this embodiment and each of the subsequent embodiments, the convolution layer and the convolution operation are denoted as Conv.

[0070] Specifically, the feature processing module includes a convolution segmentation module, a feature extraction module and a splicing and fusion module in series, wherein the convolution segmentation module is composed of a 1x1 convolution layer Conv, an activation function LeakyReLu and a segmentation layer Split; the feature extraction module includes a convolution residual module and a self-attention visual state space module in parallel; the convolution residual module is composed of a plurality of convolution residual sub-modules RB in series, and the self-attention visual state space module is composed of a first residual sub-module and a second residual sub-module in series, the first residual sub-module includes a window multi-head self-attention sub-module WMSA and a first normalization layer LayerNom, and the second residual sub-module includes a visual state space sub-module VSSM and a second normalization layer LayerNom; the splicing and fusion module is composed of a splicing layer Concat and a convolution fusion layer Conv in series.

[0071] Based on the above structure, the steps of the deep feature extraction model for feature extraction of the shallow feature in this embodiment include:

[0072] inputting the shallow feature into the feature processing module for feature extraction, and inputting the extracted shallow feature into the next feature processing module until the shallow feature extracted by the last feature processing module is obtained;

[0073] performing convolution operation on the shallow feature extracted to obtain a deep feature;

[0074] wherein the step of inputting the shallow feature into the feature processing module for feature extraction includes:

[0075] inputting the shallow feature into the convolution segmentation module for average segmentation to obtain a first shallow feature and a second shallow feature;

[0076] inputting the first shallow feature into the convolution residual module to obtain a local feature;

[0077] inputting the second shallow feature into the self-attention visual state space module to obtain a global feature;

[0078] inputting the local feature and the global feature into the splicing and fusion module to obtain a shallow feature extracted.

[0079] In the embodiment, the shallow feature is denoted as F, and the size is represented as CxH F xW F , wherein C represents the length, H F represents the height, and W F represents the width. F is first input to a convolutional segmentation module, the convolutional kernel size of the convolutional layer in the module is 1x1, and the output channel number thereof is also C, and the size is CxH F xW F . Then, F is evenly segmented into two features, denoted as F cnn and F tm , and the size of each is C / 2xH F xW F . The advantage of this is that, on the one hand, the calculation amount of the model is effectively reduced by reducing the channel number, and on the other hand, the two features are independently and in parallel processed subsequently, so that the features can be effectively extracted.

[0080] For the first shallow extracted feature F cnn , a convolutional residual module is used for convolutional operation to extract local features. In the embodiment, the convolutional residual module is composed of a plurality of convolutional residual sub-modules RB connected in series, and preferably three convolutional residual sub-modules can be set. The structure of the convolutional residual sub-module is shown in Figure 5 . Each RB is composed of two 1x1 convolutional kernels, one 3x3 convolutional kernel, and an activation function GELU. After the convolutional operation on the input first shallow extracted feature F cnn , residual connection is performed, so as to accelerate the convergence. The feature obtained after the convolutional residual module is a local feature F' cnn .

[0081] For the second shallow extracted feature F tm , a self-attention visual state space module is used for processing. First, a first residual sub-module is used for multi-head attention calculation. In the embodiment, the structure of the window multi-head self-attention sub-module WMSA in the first residual sub-module is shown in Figure 6 . In the figure, Multi-Head Attention represents multi-head attention, and ScaledDot-Product Attention represents scaled dot product attention. The core idea of WMSA is to divide the feature map into non-overlapping windows and independently perform multi-head self-attention calculation in each window. First, the input F tm is reshaped from (C / 2xH F xW F ) to:

[0082]

[0083] wherein M represents the window size.

[0084] The reshaped sequence is then divided into a plurality of non-overlapping windows, each window having a size of M, thereby obtaining N windows having a shape of (M, M, C / 2). N can be expressed as:

[0085]

[0086] For each window W i , multi-head attention calculation is independently performed. That is, after linear projection of the QKV three matrices through a linear layer Liner, inner product MatMul of Q and K, scaling and translation Scale, mask operation Mask, and normalization SoftMax are performed, and then inner product MatMul of V is performed, the outputs of each head are spliced Concat and a linear transformation Liner is performed to obtain the output of the window. Finally, the outputs of all windows are spliced to obtain the final output sequence, that is, the output is reshaped to (C / 2, H F , W F ), and then the output of the WMSA and the LayerNorm layer is connected in residual connection with the initial input to obtain the output F t .

[0087] WMSA calculates the attention mask between different windows, but does not consider the relationship between windows. In order to model the pixel relationship between windows, in the embodiment, the result F t is input to a second residual sub-module, and a second modeling of the spatial dimension of the long-range dependency relationship is performed through a visual state space sub-module VSSM and a second normalization layer LayerNorm. VSSM converts a 2D image into a 1D input sequence by using scanning in four directions, and uses a discretized iterative formula for modeling, thereby efficiently capturing global information.

[0088] In the embodiment, the structure of VSSM is as shown in Figure 7 , for the input F t , it is reshaped to (H F , W F , C / 2), denoted as F m , F m will be sent to two parallel branches for processing:

[0089] Branch 1:

[0090] Feature F mThe number of feature channels is expanded to λC in the liner layer Liner. The expanded features enter the depthwise convolutional layer DWConv. The output of DWConv enters the SiLU activation function, and then enters the 2D state space module 2DSSM to scan the 2D feature map along four different directions (top-left to bottom-right, bottom-right to top-left, top-right to bottom-left, bottom-left to top-right) to flatten the features into 1D sequences and convert the 2D image into 1D input sequences. The iteration principle is shown in Figure 8 For each 1D sequence, a discrete state space equation is used to capture long-range dependencies. The sequences obtained by scanning in four directions are summed and combined. The combined sequence is reshaped into a 2D feature map. Finally, the normalization layer is input to normalize the features, and the output is denoted as F m1 .

[0091] Branch 2:

[0092] Feature F m enters the liner layer to expand the number of feature channels to λC. The expanded features enter the SiLU activation function, and the output is denoted as F m2 .

[0093] F m1 . m2 Element-level multiplication is performed on F m and F m to fuse the information of the two branches. The fused features enter the linear layer to project the number of channels back to C / 2. The final output is F′ m1 . m has the same shape as the input feature F m2 . m .

[0094] The mathematical formula of the above process is expressed as:

[0095] F m1 =LN(2D-SSM(SiLU(DWConv(Linear(F m )))))

[0096] F m2 =SiLU(Linear(F m ))

[0097] F′ m =Linear(F m1 ⊙F m2 )

[0098] For the output F′ m of VSSM, it is normalized again by the second normalization layer LayerNorm, and the output is the global feature F′ tm .

[0099] Finally, F′ tm and F′ cnnConcat, the result is fused by a 1x1 convolution kernel Conv to obtain a low-resolution feature map containing rich information, i.e. shallow layer extraction feature.

[0100] The above steps are feature extraction steps of a feature processing module. In the embodiment, the deep layer feature extraction model includes a plurality of serially connected feature processing modules. After being processed by the plurality of feature processing modules, a convolution operation is performed to generate new deeper layer feature representation. That is, through the processing of the plurality of feature processing modules and the convolution layer, the shallow layer feature is converted into a comprehensive description of the deep layer feature of the original image, thereby obtaining the deep layer feature.

[0101] In the embodiment, the window multi-head self-attention and the visual state space model are combined to enable the deep layer feature extraction model to have efficient pixel dependency capturing capability. The window multi-head self-attention reduces the computational complexity and improves the computational efficiency by performing self-attention operation in a local window. The feature loss problem that may occur in the deep layer feature extraction process is effectively alleviated through the combination of multiple convolution residuals. Through the structural design of the deep layer feature extraction model in the embodiment, the global context modeling capability of the model is improved, and the accuracy and stability in complex image processing tasks are ensured.

[0102] After the shallow layer feature and the deep layer feature are extracted by the above steps, the shallow layer feature and the deep layer feature are fused, and the obtained image feature is input into an image enhancement model for step-by-step upsampling to reconstruct a high-quality image. In the embodiment, the structure of the image enhancement model is as shown in Figure 9 The image enhancement model includes a first convolution layer, a plurality of pixel enhancement modules PEB, and a convolution module. The pixel enhancement modules PEB and the convolution module are connected in series, and the pixel enhancement modules PEB are connected in parallel with the first convolution layer. The pixel enhancement module includes a second convolution layer, a third convolution layer connected in parallel with the second convolution layer, a pixel reorganization module pixel shuffle, and a fourth convolution layer. The third convolution layer, the pixel reorganization module pixel shuffle, and the fourth convolution layer are connected in series.

[0103] Based on the above structure, the specific steps of image reconstruction for the input image feature in the embodiment include:

[0104] The image feature is input into the pixel enhancement module for upsampling processing, and the obtained upsampling feature is input into the next pixel enhancement module until the upsampling feature output by the last pixel enhancement module is obtained.

[0105] The image feature is input into the first convolutional layer for convolution operation, and then is connected with the up-sampling feature in residual connection, and the connected feature is input into the convolutional module for convolution operation, to obtain a high-resolution image corresponding to the low-resolution image.

[0106] In the embodiment, preferably, three pixel enhancement modules PEB are provided and are connected in series. For the input image feature Feature, after up-sampling processing by the three PEBs, the image feature is connected with the initially input image feature in residual connection, wherein the specific steps of up-sampling by each PEB include:

[0107] The image feature is up-sampled by the third convolutional layer, the pixel reorganization module and the fourth convolutional layer to obtain first up-sampling feature;

[0108] The image feature is convoluted by the second convolutional layer to obtain first convolutional feature;

[0109] The first up-sampling feature and the first convolutional feature are connected in residual connection to obtain up-sampling feature.

[0110] In the embodiment, the PEB module is composed of multiple convolutional layers and a pixel reorganization module pixel shuffle. The input image feature is convoluted, up-sampled by pixel shuffle and convoluted in the PEB module, and then is connected with the input image feature in residual connection to obtain complete up-sampling operation of one PEB module. The image feature is up-sampled by three series-connected PEB modules, and then is connected with the input image feature in residual connection to obtain complete up-sampling feature. The convolution operation in the residual connection is to ensure that the image size of the input and the output is consistent.

[0111] The core principle of pixel shuffle is to rearrange the channel dimension of the feature map to the spatial dimension, so as to realize the improvement of the resolution. For the input low-resolution feature map, assuming that the size is HxWxC, wherein H is the height, W is the width, and C is the number of channels, first, the input feature channel C is expanded, and the number of expanded channels satisfies C' = C*r 2 , then the feature map is rearranged into a high-resolution feature map High-quality image with a size of r*Hxr*WxC.

[0112] For the input feature Figure X and the output feature map Y, the pixel shuffle operation can be represented as:

[0113]

[0114] where i and j are spatial indices of the output feature map, and c is a channel index, denotes a floor operation, mod denotes a modulo operation, and r is an up-sampling factor.

[0115] After the residual connection and the up-sampling, the up-sampling result is subjected to a convolution operation by a convolution module. In this embodiment, the convolution module is a convolution layer with two 3x3 convolution kernels, i.e., two consecutive convolution operations are performed using 3x3 convolution kernels, so as to improve the details of the image layer by layer.

[0116] The image enhancement model of this embodiment can efficiently up-sample the input image features, use multi-level convolution design and residual connection, and perform an end-to-end processing process, effectively solving the problem of detail loss in the image reconstruction process and ensuring that the reconstructed and enhanced high-quality image has higher resolution, clearer details and less noise.

[0117] The image super-resolution method provided in this embodiment solves the problems of unstable repair effect and insufficient detail reconstruction capability of the traditional method. The multi-scale adaptive shallow feature extraction method based on dynamic convolution can extract shallow features in multiple scales. The deep feature extraction model of the visual state space based on window multi-head self-attention combines global features and local features, significantly improves the accuracy and depth of feature extraction, and combines local feature extraction and global context modeling, and uses an image reconstruction enhancement module for high-quality reconstruction, so that the reconstructed image can present better details and overall effect. The present application effectively improves the detail clarity and image resolution of the reconstructed image, and is suitable for various complex scenes, and has good practicability.

[0118] Please refer to Figure 10 , based on the same inventive concept, the second embodiment of the present application provides an image super-resolution system, comprising:

[0119] The shallow feature extraction module 10 is configured to sample the low-resolution image according to a sampling strategy based on the region complexity to obtain shallow features.

[0120] The deep feature extraction module 20 is configured to input the shallow features into a pre-established deep feature extraction model to obtain deep features, and the deep feature extraction model is constructed based on a visual state space model with self-attention.

[0121] The feature fusion module 30 is configured to fuse the shallow features and the deep features to obtain image features.

[0122] The image reconstruction module 40 is configured to input the image features into a pre-established image enhancement model to perform upsampling and residual connection, so as to obtain a high-resolution image corresponding to the low-resolution image, and the image enhancement model is constructed by using a convolution-based pixel recombination method.

[0123] The technical features and technical effects of the image super-resolution system are the same as those of the method, which will not be repeated here. The above-mentioned modules in the image super-resolution system can be realized by software, hardware and their combinations. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0124] In addition, the embodiment of the present application also provides a computer device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the steps of the above-mentioned method.

[0125] Please refer to Figure 11 , an embodiment of the internal structure diagram of the computer device, which can be a terminal or a server. The computer device includes a processor, a memory, a network interface, a display and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to realize the image super-resolution method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0126] Those skilled in the art can understand Figure 11 that the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0127] In addition, the embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to realize the steps of the above-mentioned method.

[0128] To sum up, the image super-resolution method, system, computer device and storage medium provided by the embodiment of the present application, the method samples the low-resolution image according to the sampling strategy based on the region complexity to obtain the shallow feature; the shallow feature is input into the pre-established deep feature extraction model to obtain the deep feature, and the deep feature extraction model is constructed by using the visual state space model based on self-attention; the shallow feature and the deep feature are fused to obtain the image feature; the image feature is input into the pre-established image enhancement model for upsampling and residual connection to obtain the high-resolution image corresponding to the low-resolution image, and the image enhancement model is constructed by using the pixel recombination method based on convolution. The multi-scale adaptive shallow feature extraction method based on dynamic convolution can extract the shallow feature in multiple scales, the deep feature extraction model based on the window multi-head self-attention visual state space combines the global feature and the local feature, significantly improves the accuracy and depth of feature extraction, and combines local feature extraction and global context modeling, and uses the image reconstruction enhancement module for high-quality reconstruction, so that the reconstructed image can present better details and overall effect. The present application effectively improves the detail clarity and image resolution of the reconstructed image, and is suitable for various complex scenes, and has good practicability.

[0129] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, the system embodiment is basically similar to the method embodiment, so the description is relatively simple, and the related parts can be referred to the part of the method embodiment. It should be noted that, the technical features of the above embodiments can be combined arbitrarily, in order to make the description simple, not all possible combinations of the technical features of the above embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the description.

[0130] The above-described embodiments only express several preferred embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent. It should be noted that, for ordinary skilled in the art, without departing from the technical principles of the present application, some improvements and replacements can be made, and these improvements and replacements should be considered as the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the protection scope of the claims.

Claims

1. An image super-resolution method, characterized by, The method comprises the following steps: According to the sampling strategy based on the complexity of the region, the low-resolution image is sampled to obtain the shallow feature; The shallow feature is input into the pre-established deep feature extraction model to obtain the deep feature, and the deep feature extraction model is constructed by using the self-attention-based visual state space model; The shallow feature and the deep feature are fused to obtain the image feature; The image feature is input into the pre-established image enhancement model for upsampling and residual connection to obtain the high-resolution image corresponding to the low-resolution image, and the image enhancement model is constructed by using the convolution-based pixel reorganization method; The deep feature extraction model is composed of a plurality of feature processing modules and a convolution layer in series, and the feature processing module is constructed by using the self-attention-based visual state space model; The feature processing module comprises a convolution segmentation module, a feature extraction module and a splicing and fusion module connected in series, and the feature extraction module comprises a convolution residual module and a self-attention visual state space module connected in parallel; The convolution residual module is composed of a plurality of convolution residual submodules connected in series, and the convolution residual submodule comprises three convolution kernels of a predetermined size and an activation function; The self-attention visual state space module is composed of a first residual submodule and a second residual submodule connected in series, the first residual submodule comprises a window multi-head self-attention submodule and a first normalization layer, and the second residual submodule comprises a visual state space submodule and a second normalization layer; The step of inputting the shallow feature into the pre-established deep feature extraction model to obtain the deep feature comprises: The shallow feature is input into the feature processing module for feature extraction, and the extracted shallow feature is input into the next feature processing module until the shallow feature output by the last feature processing module is obtained; The shallow feature is input into the convolution layer for convolution operation to obtain the deep feature; The step of inputting the shallow feature into the feature processing module for feature extraction comprises: The shallow feature is input into the convolution segmentation module for average segmentation to obtain a first shallow feature and a second shallow feature; The first shallow feature is input into the convolution residual module to obtain a local feature; The second shallow feature is input into the self-attention visual state space module to obtain a global feature; The local feature and the global feature are input into the splicing and fusion module to obtain the shallow feature.

2. The image super-resolution method of claim 1, wherein, The step of sampling the low-resolution image according to the sampling strategy based on the complexity of the region to obtain the shallow feature comprises: The low-resolution image is divided into a plurality of image regions according to a predetermined size, and the gradient of each pixel point in each image region is calculated according to an edge detection algorithm; The gradient of each pixel point calculated is binarized based on a gradient threshold to obtain a binarized gradient; The proportion of pixels in each image region whose binarized gradient is a preset gradient is calculated, and the complexity type of each image region is determined according to a comparison result of the proportion and a proportion threshold, wherein the complexity type includes low complexity and high complexity. According to the complexity type of each image region, a corresponding size of a convolution kernel is selected for sampling, and the obtained sampling result is spliced and fused to obtain the shallow feature of the low-resolution image.

3. The image super-resolution method of claim 1, wherein, The image enhancement model comprises a first convolution layer, a plurality of pixel enhancement modules and a convolution module, the pixel enhancement modules and the convolution module are connected in series, and the pixel enhancement modules are connected in parallel with the first convolution layer. The pixel enhancement module comprises a second convolution layer, a third convolution layer, a pixel reorganization module and a fourth convolution layer connected in series and connected in parallel with the second convolution layer.

4. The image super-resolution method of claim 3, wherein, The step of inputting the image feature into the pre-established image enhancement model for upsampling and residual connection to obtain the high-resolution image corresponding to the low-resolution image comprises: The image feature is input into the pixel enhancement module for upsampling processing, and the obtained upsampling feature is input into the next pixel enhancement module until the upsampling feature output by the last pixel enhancement module is obtained; After the image feature is input into the first convolution layer for convolution operation, the residual connection is performed between the upsampling feature and the convolution operation, and the connected feature is input into the convolution module for convolution operation to obtain the high-resolution image corresponding to the low-resolution image.

5. The image super-resolution method of claim 4, wherein, The step of inputting the image feature into the pixel enhancement module for upsampling processing comprises: The third convolution layer, the pixel reorganization module and the fourth convolution layer are used to perform upsampling processing on the image feature to obtain first upsampling feature; The second convolution layer is used to perform convolution operation on the image feature to obtain first convolution feature; The first upsampling feature and the first convolution feature are connected in residual connection to obtain upsampling feature.

6. An image super-resolution system, the system is applied to the method of any one of claims 1 to 5, characterized in that, It comprises: A shallow feature extraction module is configured to sample a low-resolution image according to a region complexity-based sampling strategy to obtain a shallow feature. A deep feature extraction module is configured to input the shallow feature into a pre-established deep feature extraction model to obtain a deep feature, wherein the deep feature extraction model is constructed based on a self-attention-based visual state space model. A feature fusion module is configured to fuse the shallow feature and the deep feature to obtain an image feature. An image reconstruction module is configured to input the image feature into a pre-established image enhancement model for upsampling and residual connection to obtain a high-resolution image corresponding to the low-resolution image, wherein the image enhancement model is constructed based on a convolution-based pixel reorganization method.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images