Image super-resolution processing method and device, computer equipment and medium

Abstract

The invention relates to an image super-resolution processing method and device, computer equipment and a medium, and the method comprises the steps: obtaining a to-be-processed low-resolution image; extracting features of the low-resolution image to obtain an initial feature map; extracting global channel structure features in the initial feature map based on a channel attention mechanism, and extracting local channel detail features in the initial feature map based on a dense block channel attention mechanism; fusing the global channel structural features and the local channel detail features to obtain fused features, and extracting local space detail features based on the fused features and a space attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local space information; and performing up-sampling and down-sampling on the target feature map to generate a high-resolution image. According to the whole scheme, image super-resolution processing can be efficiently and accurately realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image super-resolution processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] Image super-resolution processing is the process of reconstructing a corresponding high-resolution image from one or more low-resolution images. It is a classic problem in computer vision. Image super-resolution itself is an ill-posed problem. For a low-resolution image, there are always multiple high-resolution images corresponding to it. Traditional image super-resolution algorithms are mainly divided into interpolation-based methods and learning-based methods. They use sparse coding, multi-scale and other prior knowledge modeling to achieve high-resolution image restoration. Although these algorithms can improve image resolution, they are insufficient in restoring high-frequency detail information and texture details.

[0003] With the resurgence of artificial intelligence technology and the rapid development of deep neural networks, deep neural networks have achieved great success in the field of computer vision. Image super-resolution algorithms based on deep neural networks have been actively explored, and a large number of image super-resolution models based on convolutional neural networks, generative adversarial networks, and residual neural networks have emerged. By utilizing the powerful feature representation and learning nonlinear transformation capabilities of deep neural networks, the model can more deeply explore the mapping relationship between high-resolution and low-resolution images and better reconstruct the high-frequency detail information of the image.

[0004] Although image super-resolution algorithms based on convolutional neural networks have achieved good performance, due to the limitation of the receptive field size of the convolution kernel, the neural network cannot model the dependencies between long-distance pixels in the image, and the final calculation results are inaccurate. In addition, image super-resolution algorithms based on convolutional neural networks usually require a lot of computing resources, resulting in long training time and slow algorithm tuning. Summary of the invention

[0005] Based on this, it is necessary to provide an efficient and accurate image super-resolution processing method, device, computer equipment, storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides an image super-resolution processing method. The method comprises:

[0007] Obtain a low-resolution image to be processed;

[0008] Extracting features of the low-resolution image to obtain an initial feature map;

[0009] Extracting global channel structure features in the initial feature map based on a channel attention mechanism, and extracting local channel detail features in the initial feature map based on a dense block channel attention mechanism;

[0010] The global channel structure feature and the local channel detail feature are fused to obtain a fused feature, and local spatial detail features are extracted based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information;

[0011] The target feature map is upsampled and downsampled to generate a high-resolution image.

[0012] In one embodiment, extracting the features of the low-resolution image to obtain an initial feature map includes:

[0013] Based on a convolution layer with a preset convolution kernel size, features of the low-resolution image are extracted to obtain an initial feature map.

[0014] In one embodiment, extracting the global channel structure features in the initial feature map based on the channel attention mechanism includes:

[0015] Using the initial feature map as feature embedding;

[0016] Performing matrix mapping and dimension transformation processing on the feature embedding to obtain a mapped matrix;

[0017] Based on the mapped matrix, scaled dot product attention and attention weighting processing are performed to obtain scaled dot product attention on the channel dimension, where the scaled dot product attention on the channel dimension represents the global channel structure characteristics.

[0018] In one embodiment, extracting local channel detail features in the initial feature map based on a dense block channel attention mechanism includes:

[0019] Using the initial feature map as feature embedding;

[0020] Dividing the feature embedding into a plurality of non-overlapping windows according to the dense window to form a dense window feature embedding;

[0021] Calculate channel attention for each dense window feature embedding and obtain the attention score matrix;

[0022] Normalize each row of the attention score matrix to obtain the weight matrix;

[0023] Based on the weight matrix and the attention score matrix, obtain the scaled dot product channel attention of the dense block;

[0024] The scaled dot product channel attention of each dense block is reconnected in the order of partitioning to obtain the local channel detail features.

[0025] In one embodiment, the fusing of the global channel structure feature and the local channel detail feature to obtain a fused feature, and extracting a local spatial detail feature based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information includes:

[0026] Alternately using channel attention and dense block channel attention to fuse the global channel structure feature and the local channel detail feature to obtain a fused feature;

[0027] Based on the fusion features and spatial attention mechanism, local spatial detail features are extracted to obtain the target feature map with enhanced global channel information, local channel information and local spatial information.

[0028] In one embodiment, the alternate use of channel attention and dense block channel attention to fuse the global channel structure feature and the local channel detail feature to obtain the fused feature includes:

[0029] Using the initial feature map as feature embedding;

[0030] Embedding the feature through normalization and channel attention to obtain a first output feature;

[0031] Embedding the feature and adding the first output feature to obtain an intermediate result;

[0032] The intermediate result is sent to normalization and GMLP to obtain the second output feature of channel attention;

[0033] The second output feature plus the intermediate result is used as the final output feature of the channel attention;

[0034] The final output feature is used as a dense block channel attention input to fuse the global channel structure feature and the local channel detail feature to obtain a fused feature.

[0035] In a second aspect, the present application also provides an image super-resolution processing device. The device comprises:

[0036] A data acquisition module, used for acquiring a low-resolution image to be processed;

[0037] An initialization module, used to extract features of the low-resolution image to obtain an initial feature map;

[0038] A feature extraction module, used to extract global channel structure features in the initial feature map based on a channel attention mechanism, and to extract local channel detail features in the initial feature map based on a dense block channel attention mechanism;

[0039] A feature fusion module is used to fuse the global channel structure feature and the local channel detail feature to obtain a fused feature, and extract the local spatial detail feature based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information;

[0040] The sampling module is used to upsample and downsample the target feature map to generate a high-resolution image.

[0041] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0042] Obtain a low-resolution image to be processed;

[0043] Extracting features of the low-resolution image to obtain an initial feature map;

[0044] Extracting global channel structure features in the initial feature map based on a channel attention mechanism, and extracting local channel detail features in the initial feature map based on a dense block channel attention mechanism;

[0045] The global channel structure feature and the local channel detail feature are fused to obtain a fused feature, and local spatial detail features are extracted based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information;

[0046] The target feature map is upsampled and downsampled to generate a high-resolution image.

[0047] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0048] Obtain a low-resolution image to be processed;

[0049] Extracting features of the low-resolution image to obtain an initial feature map;

[0050] Extracting global channel structure features in the initial feature map based on a channel attention mechanism, and extracting local channel detail features in the initial feature map based on a dense block channel attention mechanism;

[0051] The global channel structure feature and the local channel detail feature are fused to obtain a fused feature, and local spatial detail features are extracted based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information;

[0052] The target feature map is upsampled and downsampled to generate a high-resolution image.

[0053] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0054] Obtain a low-resolution image to be processed;

[0055] Extracting features of the low-resolution image to obtain an initial feature map;

[0056] Extracting global channel structure features in the initial feature map based on a channel attention mechanism, and extracting local channel detail features in the initial feature map based on a dense block channel attention mechanism;

[0057] The global channel structure feature and the local channel detail feature are fused to obtain a fused feature, and local spatial detail features are extracted based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information;

[0058] The target feature map is upsampled and downsampled to generate a high-resolution image.

[0059] The above-mentioned image super-resolution processing method, device, computer equipment, storage medium and computer program product obtain a low-resolution image to be processed; extract the features of the low-resolution image to obtain an initial feature map; extract the global channel structure features in the initial feature map based on the channel attention mechanism, and extract the local channel detail features in the initial feature map based on the dense block channel attention mechanism; fuse the global channel structure features and the local channel detail features to obtain a fused feature, and extract the local spatial detail features based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information; upsample and downsample the target feature map to generate a high-resolution image. In the whole process, the attention range is applied to the channel dimension, thereby realizing global attention on the high-resolution image. In addition, the image is divided into blocks in the spatial dimension, which better integrates the pixel information in the spatial dimension and the channel dimension. At the same time, the attention range is narrowed to the window after the block, which further reduces the computational complexity and can efficiently and accurately realize image super-resolution processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A diagram showing an application environment of an image super-resolution processing method in an embodiment;

[0061] Figure 2 Schematic diagram of a process flow of an image super-resolution processing method in one embodiment;

[0062] Figure 3 A schematic diagram of the overall network architecture of an image super-resolution processing method in an application example;

[0063] Figure 4 Schematic diagram of dense block channel attention;

[0064] Figure 5 is a schematic flow chart of an image super-resolution processing method in another embodiment;

[0065] Figure 6 Schematic diagram of alternating processing of channel attention and dense block channel attention during feature fusion;

[0066] Figure 7 is a structural block diagram of an image super-resolution processing device in one embodiment;

[0067] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] In order to explain in detail the technical solution of the image super-resolution processing method of the present application, the following will first introduce the Spatial Attention (SA) mechanism.

[0070] Spatial Attention (SA) mechanism. The traditional mainstream image super-resolution method is to apply multi-head self-attention (MSA) in the spatial dimension, and the spatial attention mechanism is realized by applying multi-head self-attention in the window.

[0071] Given an input image of size H×W×3, the image passes through a convolution layer with a convolution kernel size of 3*3 to form a feature embedding of size H×W×C. The input of spatial attention is the feature embedding, which is divided into M×M non-overlapping local windows, further forming a dimension of The local feature embedding is then used to calculate the self-attention for each local feature embedding. Where P Q , P K , P V Representing the mapping matrix, the calculation process of Q, K, and V represented by the query, key, and value matrices is as follows:

[0072] Q = XP Q , K = XP K , V = XP V ,

[0073] Given a dimension is The input is first divided into A local window of size M*M is created, and then attention is calculated independently in each window. Based on the spatial attention of the window (Windo w -based Multi-head Self-Attention, W-MSA) calculation process can be expressed as:

[0074]

[0075] in Corresponding to local window feature embedding Query, key, value after linear mapping, d represents the dimension of query / key, and B represents the relative position encoding. T The generated dimensions are Attention map of After scaling by the scaling factor and adding the relative position code B, a softmax function is applied to obtain the weights of these values, and finally multiplied by V to get the output of Attention(Q, K, V). The output of Attention(Q, K, V) is a scaled dot product attention of size H×W×C. Since the attention is calculated independently in each window, QK T The generated attention map is based on a spatial window with a window size of M×M, so we get a scaled dot product attention in the spatial dimension, that is, the output is a feature map of size H×W×C.

[0076] Usually in practice, according to the multi-head self-attention method in , the attention function is executed h times in parallel and the results are merged into multi-head self-attention, thereby applying multi-head self-attention in the window to implement the spatial attention mechanism. The computational complexity of window-based spatial attention is expressed as:

[0077] Ω(W-MSA)=4HWC 2 +2M 2 HWC2

[0078] Where M represents the sliding window size. Compared with MSA, limiting the attention window makes the computational complexity grow linearly with the spatial resolution, eliminating part of the limitation of the complexity growing with the quadratic HW, which effectively reduces the attention computation cost under high-resolution images.

[0079] Based on the above principles, the image super-resolution processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 sends an image super-resolution processing request to the server 104, in which the image super-resolution processing request carries a low-resolution image to be processed. The server 104 receives and parses the image super-resolution processing request to obtain the low-resolution image to be processed; extracts the features of the low-resolution image to obtain an initial feature map; extracts the global channel structure features in the initial feature map based on the channel attention mechanism, and extracts the local channel detail features in the initial feature map based on the dense block channel attention mechanism; fuses the global channel structure features and the local channel detail features to obtain a fused feature, and extracts the local spatial detail features based on the fused features and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information; upsamples and downsamples the target feature map to generate a high-resolution image. Further, the server 104 can feed back the generated high-resolution image to the terminal 102. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0080] In one embodiment, Figure 2 As shown, a method for image super-resolution processing is provided, and the method is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0081] S100: Acquire a low-resolution image to be processed.

[0082] The low-resolution image to be processed refers to the low-resolution image that needs to be processed this time. Specifically, the low-resolution image may be an input image of a size of H×W×3.

[0083] S200: Extract features of the low-resolution image to obtain an initial feature map.

[0084] The features of the low-resolution image obtained in S100 are extracted to obtain an initial feature map. Specifically, features can be extracted through a convolution layer to obtain an initial feature map, and the initial feature map is used as a feature embedding for subsequent processing. In practical applications, the low-resolution image can be passed through a convolution layer with a preset convolution kernel size to extract the features of the low-resolution image and obtain an initial feature map. Furthermore, the convolution layer with a preset convolution kernel size is a convolution layer with a convolution kernel size of 3*3. After the input image of H×W×3 passes through the convolution layer with a convolution kernel size of 3*3, a feature embedding of size H×W×C is formed.

[0085] S300: extracting global channel structure features in the initial feature map based on the channel attention mechanism, and extracting local channel detail features in the initial feature map based on the dense block channel attention mechanism.

[0086] Here, based on the channel attention mechanism (Channel Attention Mechanism) and the dense block channel attention mechanism (Dens e The Block Channel Attention Mechanism is used to extract the global channel structure features and local channel detail features in the initial feature map. The channel attention mechanism focuses on the channel dimension of the feature map and enhances the representation ability of the network by reweighting the feature response of each channel. The dense block channel attention mechanism is based on the channel attention mechanism and extracts more detailed local channel features by introducing local dense connections and block processing.

[0087] S400: Fusing global channel structure features and local channel detail features to obtain fused features, and extracting local spatial detail features based on the fused features and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information.

[0088] The global channel structure features and local channel detail features are concatenated in the channel dimension. In this way, the feature vector at each position will contain both global channel information and local channel detail information. The concatenated features may contain redundant information or need further optimization. This information can be further fused through a convolutional layer (usually with a 1x1 convolution kernel) to learn the interactions and dependencies between different channels. The convolutional layer is usually followed by a nonlinear activation function (such as ReLU) to enhance the nonlinear expression ability of the network. The spatial attention mechanism aims to focus on important spatial locations or regions in the feature map. Unlike the channel attention mechanism, the spatial attention mechanism operates on the spatial dimension (i.e., the width and height of the feature map). Usually, a spatial attention map can be generated by convolving the fused feature map and using the Sigmoid activation function. The size of this attention map is the same as the original feature map, but the value at each position represents the importance or weight of the position. The generated spatial attention map is element-wise multiplied with the fused feature map to achieve weighted feature responses at different spatial locations. In this way, the network can pay more attention to those important spatial positions or regions, thereby extracting more refined local spatial detail features. The feature map weighted by the spatial attention mechanism is the target feature map with enhanced global channel information, local channel information and local spatial information. This feature map not only contains the global channel structure features and local channel detail features, but also further highlights the important spatial detail features through the spatial attention mechanism.

[0089] In practical applications, channel attention and dense block channel attention can be used alternately to fuse global channel structure features and local channel detail features to obtain fused features.

[0090] S500: Upsample and downsample the target feature map to generate a high-resolution image.

[0091] Upsampling is the process of increasing the resolution (i.e. width and height) of an image. This is usually achieved through interpolation methods such as bilinear interpolation, bicubic interpolation, or more complex learning-based upsampling methods (such as transposed convolution, sub-pixel convolution, etc.). Downsampling is the process of reducing the resolution of an image. This is usually achieved through pooling operations (such as maximum pooling, average pooling) or convolution with strides. Downsampling is often used in deep learning to reduce the amount of computation, increase the receptive field, or extract higher-level features. Here, upsampling and downsampling are performed on the target feature map to generate a high-resolution image.

[0092] In practical applications, the overall network architecture corresponding to the image super-resolution processing method of this application is as follows: Figure 3 shown.

[0093] The above-mentioned image super-resolution processing method obtains a low-resolution image to be processed; extracts features of the low-resolution image to obtain an initial feature map; extracts global channel structure features in the initial feature map based on a channel attention mechanism, and extracts local channel detail features in the initial feature map based on a dense block channel attention mechanism; fuses global channel structure features and local channel detail features to obtain fused features, and extracts local spatial detail features based on the fused features and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information, and local spatial information; upsamples and downsamples the target feature map to generate a high-resolution image. In the whole process, the attention range is applied to the channel dimension, thereby realizing global attention on the high-resolution image. In addition, the image is divided into blocks in the spatial dimension, which better integrates the pixel information in the spatial dimension and the channel dimension. At the same time, the attention range is narrowed to the window after the block, which further reduces the computational complexity and can efficiently and accurately realize image super-resolution processing.

[0094] In one embodiment, extracting global channel structure features in an initial feature map based on a channel attention mechanism includes: using the initial feature map as feature embedding; performing mapping matrix and dimensional transformation processing on the feature embedding to obtain a mapped matrix; performing scaled dot product attention and attention weighting processing based on the mapped matrix to obtain scaled dot product attention on the channel dimension, wherein the scaled dot product attention on the channel dimension represents the global channel structure features.

[0095] Channel Attention (CA) mechanism. The channel attention mechanism applies multi-head self-attention in the channel dimension. Given an input image of size H×W×3, the image passes through a convolution layer with a convolution kernel size of 3*3 to form a feature embedding of size H×W×C. The input of the channel attention is the feature embedding. The specific calculation process is expressed as:

[0096]

[0097] The matrix is ​​composed of feature embeddings of size H×W×C After the mapping matrix, α represents a learnable scaling parameter, which is used to control the amplitude of the dot product of K and Q, that is, the scaling factor in the corresponding spatial attention After scaling, the attention score is obtained, which can represent the channel attention score of the extracted global features and the global structural features on the channel. A softmax function is applied to obtain the weights of these values, and finally multiplied by V to obtain the output of Attention(Q, K, V). The output of Attention(Q, K, V) is a scaled dot product attention of size H×W×C. Since the dimension of K is C×H×W and the dimension of V is H×W×C, the dimension of the attention map generated by K·Q is C×C. The length and width represent the number of channels, so the scaled dot product attention on the channel dimension is obtained, that is, the output is a feature map of size H×W×C.

[0098] In practice, similar to the multi-head self-attention method, the number of channels is divided into h heads, and the method is executed h times in parallel. The attention of different channels is learned separately and the results are merged. In this way, multi-head self-attention is applied in the channel dimension to realize the channel attention mechanism. In terms of computational complexity, the biggest difference between channel attention and spatial attention is that the dimensions of the generated attention map are different, which affects the multiplication of Q by K. T And the complexity of the attention score A multiplied by V. The complexity of channel attention is expressed as:

[0099] Ω(CA)=6HWC 2

[0100] In one embodiment, extracting local channel detail features in the initial feature map based on a dense block channel attention mechanism includes:

[0101] Step 1: Use the initial feature map as feature embedding.

[0102] After convolution processing, the low-resolution image obtains an initial feature map, which is used as input, that is, the dense block channel attention mechanism is started with the initial feature map as feature embedding.

[0103] Step 2: Divide the feature embedding into several non-overlapping windows according to the dense window to form a dense window feature embedding.

[0104] The feature embedding is divided into several non-overlapping windows according to a predetermined dense window size (such as 3x3, 5x5, etc.). In this way, the original feature map is divided into multiple small blocks, and each small block (dense window feature embedding) contains the feature information of the local area.

[0105] Step 3: Calculate channel attention for each dense window feature embedding and obtain the attention score matrix.

[0106] For each dense window feature embedding, calculate its channel attention. This is usually implemented through some form of attention mechanism, such as Squeeze-and-Excitation (SE) block. The SE block first squeezes the features, compresses the feature map of each channel into a real number through global average pooling to obtain global information; then, through the excitation operation, a fully connected layer (or equivalent 1x1 convolution) is used to learn the correlation between different channels and output the weight (i.e., attention score) of each channel.

[0107] Step 4: Normalize each row of the attention score matrix to obtain the weight matrix.

[0108] Each row of the resulting attention score matrix is ​​normalized (e.g., using a softmax function) to ensure that the sum of the weights of all channels is 1. In this way, the weight of each channel represents its importance in the current dense window.

[0109] Step 5: Based on the weight matrix and the attention score matrix, get the scaled dot product channel attention of the dense block.

[0110] Based on the normalized weight matrix and the original attention score matrix, the scaled dot product channel attention of each dense window is calculated by scaling the dot product. This step actually re-weights each channel of the feature map, emphasizing important channels and suppressing unimportant channels.

[0111] Step 6: Reconstruct the scaled dot product channel attention of each dense block in the order of partitioning to obtain the local channel detail features.

[0112] Finally, the scaled dot product channel attention of each dense block is reassembled in the original partition order to form the final local channel detail feature map. This feature map not only retains the spatial information of the original feature map, but also enhances the important local channel features through the channel attention mechanism.

[0113] Through the above steps, the dense block channel attention mechanism can effectively extract more refined and discriminative local channel detail features from the initial feature map, which is of great significance for improving the performance of subsequent processing tasks (such as classification, detection, etc.).

[0114] The following will use a specific application example to explain in detail the specific implementation process of the entire dense block channel attention mechanism.

[0115] Specific as Figure 4As shown in the figure, the blocking principle of dense blocking channel attention. Dense attention allows each pixel to interact with other pixels in a non-overlapping M*M window. The dense blocking method is very similar to the sliding window mechanism, and only considers the interaction with pixels in the dense area. However, unlike the sliding window spatial attention, the dense window is introduced into the global channel attention. By changing the scope of the channel attention from the original global scope to the window scope, the model's ability to reconstruct local channel information is improved, making up for the shortcomings of Restormer in local detail recovery.

[0116] The specific implementation is as follows: given an input image of size H×W×3, the image passes through a convolution layer with a convolution kernel size of 3×3 to form a feature embedding of size H×W×C. The input of the channel attention is the feature embedding, which is divided into several non-overlapping windows in the order of dense window division. The number of windows is Each window contains M*M pixels, forming a dimension of The dense window feature embedding of , and then the channel attention is calculated for each dense window feature embedding. The dense block channel attention can be expressed as:

[0117]

[0118] They represent the query, key, and value after the linear mapping of dense window feature embedding. K·Q performs a dot product of K of size C×M×M and Q of size M×M×C, and the resulting dimension is The dense window attention score of K·Q is then dot-producted with V of size M×M×C to obtain a dense block-based scaled dot product channel attention of size M×M×C. Finally, each M×M×C channel is spliced ​​back in the original order of dense window partitioning to form a feature map of size H×W×C, which represents the scaled dot product attention based on the dense window in the channel dimension, that is, the output is a feature map of size H×W×C.

[0119] In terms of computational complexity, dense block channel attention maintains the same computational complexity as CA:

[0120]

[0121] Where M represents the size of the dense window. Although the complexity is the same as that of the channel attention, the dense window mechanism is introduced into the channel attention. The channel attention is calculated within a certain spatial range, and the spatial information is introduced into the channel attention and effectively integrated, which improves the channel information recovery ability in a small space.

[0122] like Figure 5As shown, in one embodiment, S400 includes:

[0123] S420: alternately use channel attention and dense block channel attention to fuse global channel structure features and local channel detail features to obtain fused features.

[0124] First, the basic channel attention module is used to preliminarily process the global channel structure features to enhance the importance of global features. Then, the dense block channel attention mechanism is introduced to refine the local channel detail features to extract richer local features. By using these two attention mechanisms alternately or in parallel, the global channel features and local channel features can be effectively fused to obtain fused features that contain both global structures and local details.

[0125] S440: Extract local spatial detail features based on fusion features and spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information.

[0126] The spatial attention mechanism focuses on the importance of different regions in the image. By modeling the dependencies of spatial positions, spatial attention can guide the model to focus on key areas in the image and ignore irrelevant background information. The fused features are input into the spatial attention module. The spatial attention module analyzes the importance of each pixel (or each position on the feature map) and generates a spatial attention map. This map reflects the saliency of different regions in the image. The fused features are weighted using this spatial attention map to highlight important local spatial details and suppress unimportant areas. In the end, the resulting target feature map contains not only enhanced global channel information and local channel information, but also optimized local spatial detail information, which is very beneficial for subsequent image understanding tasks.

[0127] In one embodiment, channel attention and dense block channel attention are used alternately to fuse global channel structure features and local channel detail features, and the fused features obtained include:

[0128] Step 1: Use the initial feature map as feature embedding.

[0129] Step 2: The feature embedding is normalized and channel-attentive to obtain the first output feature.

[0130] First, the feature embeddings are normalized, which helps improve the convergence and stability of the network. Then the channel attention mechanism is applied, which recalibrates the importance of each channel by modeling the dependencies between channels. This usually involves global average pooling, fully connected layers (or convolutional layers), and sigmoid activation on the feature map, and finally generates a weight vector with the same number of channels, which is used to weight each channel of the original feature map.

[0131] Step 3: Add the feature embedding to the first output feature to get the intermediate result.

[0132] The features processed by channel attention (the first output features) are added to the original feature embedding to get the intermediate result. This residual connection helps to preserve the original information and speed up training.

[0133] Step 4: Send the intermediate result to normalization and GMLP to obtain the second output feature of channel attention.

[0134] Step 5: Add the second output feature and the intermediate result as the final output feature of the channel attention.

[0135] The GMLP processed feature (second output feature) is added to the intermediate result again to get the final output feature of channel attention. This step is also the embodiment of residual connection, which helps to retain more information.

[0136] Step 6: Use the final output features as dense block channel attention input to fuse the global channel structure features and local channel detail features to obtain the fused features.

[0137] The final output features of the channel attention are used as the input of the dense block channel attention. A dense block channel attention mechanism is applied. This mechanism may involve dividing the feature map into multiple blocks and applying some form of attention mechanism (such as self-attention, cross attention, etc.) within each block or across blocks to capture more fine-grained local channel features. At the same time, it may also interact with the global channel features to achieve the fusion of global and local channel features.

[0138] Specifically, if Figure 6 As shown in the figure, the two modules of channel attention CA and the proposed dense block channel attention DW-CA are used alternately here. Here, channel attention is used to extract the global structure first, and then dense block channel attention is used to extract local details. The specific process is given an input image of size H×W×3, which passes through a convolution layer with a convolution kernel size of 3×3 to form an input feature x of size H×W×C i-1 , after layer normalization (LayerNorm, LN) and channel attention, the output feature plus the input feature x i-1Then get the intermediate result x t , x t It is sent to the layer normalization and gated MLP (Gated multi-layer perception, GMLP), and the final channel attention output x i The output of GMLP plus x t is formed and used as the input of the next dense block channel attention. Similarly, x t+1 is the output of LN and DW-CA plus x i Get, x i+1 is the output of two consecutive attention modules. The whole process can be described as:

[0139] x t =CA(LN(x i-1 ))+x i-1

[0140] x i =GMLP(LN(x t ))+x t

[0141] x t+1 =DW-CA(LN(x i ))+x i

[0142] x i-1 =GMLP(LN(x t+1 ))+x t+1 .

[0143] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0144] Based on the same inventive concept, the embodiment of the present application also provides an image super-resolution processing device for implementing the image super-resolution processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image super-resolution processing device embodiments provided below can refer to the limitations on the image super-resolution processing method above, and will not be repeated here.

[0145] In one embodiment, Figure 7 As shown, an image super-resolution processing device is provided, comprising:

[0146] A data acquisition module 100 is used to acquire a low-resolution image to be processed;

[0147] Initialization module 200, used to extract features of the low-resolution image and obtain an initial feature map;

[0148] A feature extraction module 300 is used to extract global channel structure features in the initial feature map based on a channel attention mechanism, and to extract local channel detail features in the initial feature map based on a dense block channel attention mechanism;

[0149] A feature fusion module 400 is used to fuse the global channel structure feature and the local channel detail feature to obtain a fused feature, and extract the local spatial detail feature based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information;

[0150] The sampling module 500 is used to upsample and downsample the target feature map to generate a high-resolution image.

[0151] In one of the embodiments, the initialization module 200 is further used to extract features of the low-resolution image based on a convolution layer with a preset convolution kernel size to obtain an initial feature map.

[0152] In one embodiment, the feature extraction module 300 is also used to use the initial feature map as feature embedding; perform mapping matrix and dimensional transformation processing on the feature embedding to obtain a mapped matrix; perform scaled dot product attention and attention weighting processing based on the mapped matrix to obtain scaled dot product attention on the channel dimension, and the scaled dot product attention on the channel dimension represents the global channel structure characteristics.

[0153] In one embodiment, the feature extraction module 300 is also used to use the initial feature map as feature embedding; divide the feature embedding into a number of non-overlapping windows according to the dense window to form a dense window feature embedding; calculate the channel attention for each dense window feature embedding to obtain an attention score matrix; normalize each row of the attention score matrix to obtain a weight matrix; based on the weight matrix and the attention score matrix, obtain the scaled dot product channel attention of the dense blocks; re-join the scaled dot product channel attention of each dense block in the order of division to obtain local channel detail features.

[0154] In one embodiment, the feature fusion module 400 is also used to alternately use channel attention and dense block channel attention to fuse global channel structure features and local channel detail features to obtain fused features; based on the fused features and the spatial attention mechanism, local spatial detail features are extracted to obtain a target feature map with enhanced global channel information, local channel information and local spatial information.

[0155] In one embodiment, the feature fusion module 400 is also used to use the initial feature map as feature embedding; normalize and channel-attention the feature embedding to obtain a first output feature; add the feature embedding to the first output feature to obtain an intermediate result; send the intermediate result to normalization and GMLP to obtain a second output feature of the channel attention; add the second output feature to the intermediate result as the final output feature of the channel attention; use the final output feature as the dense block channel attention input to fuse the global channel structure feature and the local channel detail feature to obtain a fused feature.

[0156] Each module in the above-mentioned image super-resolution processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0157] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image super-resolution processing method is implemented.

[0158] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned image super-resolution processing method when executing the computer program.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image super-resolution processing method is implemented.

[0161] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned image super-resolution processing method when executed by a processor.

[0162] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0163] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for image super-resolution processing, characterized in that: The method comprises: Obtain a low-resolution image to be processed; Extracting features of the low-resolution image to obtain an initial feature map; Extracting global channel structure features in the initial feature map based on a channel attention mechanism, and extracting local channel detail features in the initial feature map based on a dense block channel attention mechanism; The global channel structure feature and the local channel detail feature are fused to obtain a fused feature, and local spatial detail features are extracted based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information; The target feature map is upsampled and downsampled to generate a high-resolution image.

2. The method according to claim 1, characterized in that Extracting the features of the low-resolution image to obtain an initial feature map comprises: Based on a convolution layer with a preset convolution kernel size, features of the low-resolution image are extracted to obtain an initial feature map.

3. The method according to claim 1, characterized in that Extracting the global channel structure features in the initial feature map based on the channel attention mechanism includes: Using the initial feature map as feature embedding; Performing matrix mapping and dimension transformation processing on the feature embedding to obtain a mapped matrix; Based on the mapped matrix, scaled dot product attention and attention weighting processing are performed to obtain scaled dot product attention on the channel dimension, where the scaled dot product attention on the channel dimension represents the global channel structure characteristics.

4. The method according to claim 1, characterized in that: Extracting local channel detail features in the initial feature map based on the dense block channel attention mechanism includes: Using the initial feature map as feature embedding; Dividing the feature embedding into a plurality of non-overlapping windows according to the dense window to form a dense window feature embedding; Calculate channel attention for each dense window feature embedding and obtain the attention score matrix; Normalize each row of the attention score matrix to obtain the weight matrix; Based on the weight matrix and the attention score matrix, obtain the scaled dot product channel attention of the dense block; The scaled dot product channel attention of each dense block is reconnected in the order of partitioning to obtain the local channel detail features.

5. The method according to claim 1, characterized in that The fusing of the global channel structure feature and the local channel detail feature to obtain a fused feature, and extracting a local spatial detail feature based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information includes: Alternately using channel attention and dense block channel attention to fuse the global channel structure feature and the local channel detail feature to obtain a fused feature; Based on the fusion features and spatial attention mechanism, local spatial detail features are extracted to obtain the target feature map with enhanced global channel information, local channel information and local spatial information.

6. The method according to claim 5, characterized in that The alternate use of channel attention and dense block channel attention, fusing the global channel structure features and the local channel detail features, to obtain the fused features includes: Using the initial feature map as feature embedding; Embedding the feature through normalization and channel attention to obtain a first output feature; Embedding the feature and adding the first output feature to obtain an intermediate result; The intermediate result is sent to normalization and GMLP to obtain the second output feature of channel attention; The second output feature plus the intermediate result is used as the final output feature of the channel attention; The final output feature is used as a dense block channel attention input to fuse the global channel structure feature and the local channel detail feature to obtain a fused feature.

7. An image super-resolution processing device, characterized in that: The device comprises: A data acquisition module, used for acquiring a low-resolution image to be processed; An initialization module, used to extract features of the low-resolution image to obtain an initial feature map; A feature extraction module, used to extract global channel structure features in the initial feature map based on a channel attention mechanism, and to extract local channel detail features in the initial feature map based on a dense block channel attention mechanism; A feature fusion module is used to fuse the global channel structure feature and the local channel detail feature to obtain a fused feature, and extract the local spatial detail feature based on the fused feature and the spatial attention mechanism to obtain a target feature map with enhanced global channel information, local channel information and local spatial information; The sampling module is used to upsample and downsample the target feature map to generate a high-resolution image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images