An image restoration method and system based on multiple attention mechanism

Abstract

The present invention provides an image restoration method and system based on a multiple attention mechanism, which relates to the field of image processing technology. The method comprises: obtaining an image to be processed; extracting shallow features of the image to be processed through a convolution layer; calculating attention features through a triple attention module according to the shallow features, processing the shallow features through a feature aggregation module, and extracting deep features; fusing the attention features, deep features and shallow features to obtain fused features; restoring the image to be processed according to the fused features, and outputting the restored image. The present invention can perform effective feature fusion and selection between information of different scales and different types, thereby improving the quality and generalization ability of image restoration.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image restoration method and system based on a multiple attention mechanism. Background Art

[0002] Image restoration is an important task in the field of modern image processing and computer vision, especially in the fields of surveillance, medical imaging, remote sensing, etc. The goal of image restoration is to restore high-quality original images from degraded or damaged images. Common image degradation includes blur, noise, low resolution, etc., among which blur degradation is particularly common and complex, usually caused by factors such as camera motion, focus misalignment, and atmospheric turbulence.

[0003] With the rapid development of society, image restoration technology is becoming more and more intelligent. Through image restoration technology, we can solve the image degradation problems caused by imaging conditions, equipment performance, etc., thereby improving the clarity, resolution and availability of the image, and providing reliable data support for various image analysis and visual tasks, thereby meeting users' needs for high-quality images and improving user satisfaction.

[0004] At present, the image restoration methods for blur degradation mainly include traditional methods and deep learning-based methods. Traditional methods are usually based on physical models, such as point spread function (PSF) modeling, and perform image restoration through inverse filtering, Wiener filtering and other technologies. These methods work well under specific conditions, but in practical applications they are often limited by the lack of accuracy and robustness of the degradation model, and are difficult to deal with complex blur types. At the same time, traditional deep learning methods usually rely on a single feature extraction method and cannot effectively capture multi-scale and multi-type information in the image, resulting in poor restoration effects. With the development of deep learning, methods based on convolutional neural networks (CNN) have gradually become mainstream. Methods based on convolutional neural networks can automatically learn the mapping relationship from blurry images to clear images through large amounts of data training.

[0005] However, existing deep learning methods still face challenges in dealing with complex blur degradation. Since global context information is often ignored when dealing with local details, the restored image is distorted in details and structure. In addition, when faced with diverse blur types, the generalization ability is weak and overfitting is prone to occur. Especially in the case of multi-scale and multi-type blur degradation, the generalization ability and restoration quality of the model need to be improved. Summary of the invention

[0006] In order to solve the problem that the deep learning methods in the prior art often ignore the global context information when processing local details when dealing with complex blur degradation, resulting in distortion of details and structure of the restored image, and weak generalization ability and prone to overfitting when facing various blur types, the present invention provides an image restoration method and system based on a multiple attention mechanism.

[0007] The technical solution provided by the embodiment of the present invention is as follows:

[0008] First aspect

[0009] An embodiment of the present invention provides an image restoration method based on a multiple attention mechanism, comprising:

[0010] S1: Get the image to be processed;

[0011] S2: Extract shallow features of the image to be processed through the convolution layer;

[0012] S3: Based on the shallow features, the triple attention module is used to calculate the attention features; the shallow features are processed through the feature aggregation module to extract the deep features;

[0013] S4: Fuse the attention features, deep features and shallow features to obtain fused features;

[0014] S5: Perform image restoration on the image to be processed according to the fusion features, and output the restored image.

[0015] Second aspect

[0016] An embodiment of the present invention provides an image restoration system based on a multiple attention mechanism, comprising:

[0017] processor;

[0018] A memory having computer-readable instructions stored therein, wherein when the computer-readable instructions are executed by the processor, the image restoration method based on the multiple attention mechanism as in the first aspect is implemented.

[0019] The third aspect

[0020] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the image restoration method based on the multi-attention mechanism as in the first aspect is implemented.

[0021] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0022] In the present invention, by acquiring the image to be processed and gradually extracting shallow and deep features, combined with the triple attention module and the feature aggregation module, the image features can be comprehensively enhanced from multiple dimensions and levels. The triple attention module accurately captures key details and optimizes feature representation through the mechanisms of space, channel and global attention, while the feature aggregation module effectively fuses shallow and deep features to achieve comprehensive utilization of multi-scale information. Finally, the reconstruction step after feature fusion ensures the high quality of the restored image, making the method more robust and accurate when processing complex degraded images. By introducing a multiple attention mechanism, the method can effectively fuse and select features between information of different scales and types, thereby improving the quality and generalization ability of image restoration, enhancing the clarity, resolution and usability of the image, and providing reliable data support for various image analysis and visual tasks, thereby meeting the user's demand for high-quality images and improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of an image restoration method based on a multiple attention mechanism provided by an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of the structure of an image restoration network based on a multiple attention mechanism provided by an embodiment of the present invention;

[0026] Figure 3 A structural diagram of a feature aggregation module provided in an embodiment of the present invention;

[0027] Figure 4 A flowchart of a spatial window self-attention provided by an embodiment of the present invention;

[0028] Figure 5 A flow chart of a channel self-attention mechanism provided by an embodiment of the present invention;

[0029] Figure 6 An image processing flow based on multi-scale feature extraction and convolution operation provided by an embodiment of the present invention;

[0030] Figure 7 A structure of a convolutional neural network module provided by an embodiment of the present invention;

[0031] Figure 8A schematic diagram of the structure of an image restoration system based on a multiple attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0033] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0034] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0035] Reference Manual Attached Figure 1 , showing a flow chart of an image restoration method based on a multiple attention mechanism provided in an embodiment of the present invention.

[0036] The embodiment of the present invention provides an image restoration method based on a multiple attention mechanism, which can be implemented by an image restoration device based on a multiple attention mechanism, and the image restoration device based on a multiple attention mechanism can be a terminal or a server. The processing flow of the image restoration method based on a multiple attention mechanism may include the following steps:

[0037] S1: Obtain the image to be processed.

[0038] The images to be processed refer to low-quality or degraded images received from external input or sensors, which may have poor visual effects due to various reasons (such as blur, noise, low resolution, etc.).

[0039] It should be noted that the image to be processed is a low-quality image block generated by degrading a high-quality image (HR).

[0040] S2: Through the convolution layer, the shallow features of the image to be processed are extracted.

[0041] Among them, the convolution layer refers to a neural network layer used to extract image features in a deep learning network. It performs sliding calculations on the image through the convolution kernel (or filter) to identify and extract local features in the image, such as edges, textures, etc. Shallow features refer to low-level features extracted in the first few layers of the network. These features usually contain more edges, color changes, and basic structure information.

[0042] It should be noted that by extracting shallow features through the convolutional layer, the basic information of the image can be effectively captured, laying the foundation for subsequent deep feature extraction and attention mechanism operations. Shallow features can retain the basic texture and structural information of the original image, which is crucial in image reconstruction.

[0043] In a possible implementation manner, S2 is specifically:

[0044] Through the convolution layer, the shallow features of the image to be processed are extracted according to the following formula:

[0045]

[0046] in, Represents the shallow features of the image to be processed, represents the image to be processed, Conv 3 x 3 means that the convolution kernel size of the convolution layer is 3x3.

[0047] Reference Manual Attached Figure 2 , showing a structural schematic diagram of an image restoration network based on a multiple attention mechanism provided by an embodiment of the present invention.

[0048] Figure 2 In the figure, LR input represents the input layer, the two Convs above and below represent the convolutional layers of the first shallow feature extraction layer, TAB from left to right represents the first attention mechanism block and the second attention mechanism block, SW-SA, CW-SA and TCGA represent the spatial window attention unit, channel window attention unit and token dictionary cross global attention unit, SGFN represents the spatial gate feedforward network layer, Dysample represents the downsampling layer, HR output represents the output layer, and FAM from left to right below represents multiple feature aggregation layers.

[0049] Specifically, the image restoration network includes a first branch and a second branch, and the first branch and the second branch are jump-connected. The first branch includes an input layer, a first convolutional layer, a first multiple attention mechanism layer, a feature fusion layer, a second multiple attention mechanism layer, a second convolutional layer, a downsampling layer, a third convolutional layer and an output layer connected in sequence. The second branch includes an input layer, a fourth convolutional layer, multiple feature aggregation layers, a downsampling layer, a third convolutional layer and an output layer connected in sequence.

[0050] Reference Manual Attached Figure 3 , showing a structural schematic diagram of a feature aggregation module provided in an embodiment of the present invention.

[0051] Figure 3In the figure, Feature Aggregation Module represents the feature aggregation module, Layernorm represents the layer normalization unit, which is an operation used to normalize the features of each layer to accelerate model training and stabilize model performance, CCFM represents the channel selective feature module, SSFM represents the spatial selective feature module, and ⊕ represents the residual connection.

[0052] First, the input features are layer-normalized, and then the spatial selective feature module selects important spatial features by using the self-attention mechanism. After the spatial selective feature module is processed, a downward arrow and ⊕ are shown in the figure, indicating a residual connection. This residual connection adds the output of the spatial selective feature module to the normalized input features, thereby retaining a portion of the original input information. At the same time, the channel selective feature module mainly processes or selects channels to further improve the information interaction between channels and enhance the feature expression capability. Similarly, after the channel selective feature module, there is another residual connection operation, which is used to add the output of the channel selective feature module to the result after normalization and processing by the previous module. After all major modules and residual connections are completed, layer normalization is performed again to ensure the stability and consistency of the final feature output.

[0053] S3: Based on the shallow features, the triple attention module is used to calculate the attention features. The shallow features are processed through the feature aggregation module to extract the deep features.

[0054] Among them, the triple attention module refers to a module composed of three attention mechanisms, usually including spatial attention, channel attention and token dictionary cross-global attention, to extract features from multiple dimensions and enhance feature representation capabilities. Attention features refer to features obtained after calculation through the attention mechanism. These features can be adaptively enhanced according to the correlation of different parts of the input to capture important information. Feature aggregation module (FAM) refers to a module used to comprehensively process multi-level and multi-scale features. By fusing shallow and deep features, the expression ability of features is improved. Deep features refer to higher-level features extracted through deep convolution and complex operations, which contain more abstract image information.

[0055] It should be noted that through the design of the triple attention module (TAB) and the feature aggregation module (FAM), deep semantic information can be finely extracted and enhanced from shallow features. The multiple attention mechanism of TAB helps the model to adaptively select features from three different perspectives: space, channel, and global information, so that the model can more flexibly capture the key information in the image. At the same time, the FAM module can effectively fuse multiple layers of features, maintain the diversity and stability of features, and provide higher quality feature input for subsequent image restoration steps.

[0056] In a possible implementation, in S3, according to the shallow features, the attention is calculated by the triple attention module, specifically including:

[0057] S301: Calculate the self-attention features of the spatial window based on the spatial dimension.

[0058] S302: Calculate the self-attention features of the channel window based on the channel dimension.

[0059] S303: Calculate global attention features based on token dictionary intersection.

[0060] Reference Manual Attached Figure 4 , showing a flowchart of a spatial window self-attention provided by an embodiment of the present invention.

[0061] Figure 4 In the above figure, Spatial Window Self-Attention represents the spatial window self-attention mechanism. V s represents the projection value matrix, T represents transpose, Q s represents the projection query matrix, K s Represents the projection key matrix. The Softmax operation is used to apply the Softmax function to the calculated similarity matrix to normalize it, ensuring that the value of the similarity matrix is ​​between 0 and 1. Y s Represents the feature of spatial window self-attention, Reshape represents reshaping, and NwxNw represents the size of pixels.

[0062] The input projection value matrix row is linearly transformed to generate the projection query matrix and the projection key matrix. The dot product is performed on the transpose of the projection query matrix and the projection key matrix. The similarity between each pixel and other pixels is calculated. The Softmax function is applied to the calculated similarity matrix for normalization to ensure that the value of the similarity matrix is ​​between 0 and 1. The projection value matrix is ​​weighted and summed to generate the spatial self-attention feature. Finally, the spatial self-attention feature is reshaped to the same size H×W×C as the input feature so that it can be combined with subsequent features.

[0063] In a possible implementation, S301 specifically includes:

[0064] S3011: Generate the query matrix, key matrix and value matrix of the shallow features of the image to be processed according to the spatial window self-attention of the spatial dimension:

[0065]

[0066] in, Q represents the query matrix, K represents the key matrix, V represents the value matrix, X Represents shallow features, W Q , W K and W V Represents a linear projection.

[0067] S3012: Divide the query matrix, key matrix, and value matrix into non-overlapping windows respectively, and flatten the windows.

[0068] Among them, the query matrix is ​​a core component of the attention mechanism, which is used to represent the query information of the input features. In the attention calculation, the query matrix is ​​used to match the key matrix to measure the relevance of a specific position. The key matrix is ​​another component of the attention mechanism, which contains reference information of all potential matches. The attention mechanism calculates the similarity by comparing the query matrix and the key matrix to determine the weight of each position in the output. The value matrix is ​​the third core part of the attention mechanism, which contains the value information of the input features. After calculating the attention weights, the model performs weighted summation on the value matrix according to these weights to generate the final output features.

[0069] S3013: According to the flattened window, the query matrix, key matrix and value matrix are divided into h Size:

[0070] in, Q s , K s and V s denote the projection query matrix, projection key matrix and projection value matrix respectively, Indicates i The query matrix in the attention head, Indicates i The key matrix in the attention head, Indicates i The value matrix in the attention head, , h represents the total number of attention heads, and the size of each attention head is , c represents the dimension of the channel, Indicates i The output of the attention head, softmax represents the activation function, T represents transpose, BRelative position encoding.

[0071] S3014: Concatenate the outputs of each attention head:

[0072]

[0073] in, Y s Represents the characteristics of spatial window self-attention, concat Represents a splicing operation, h represents the total number of attention heads, SW - SA represents the spatial window self-attention operation, W P Represents the linear projection that fuses all features.

[0074] It should be noted that the model can effectively capture the spatial relationships and features in the image through the self-attention mechanism in the spatial dimension. These steps start with generating query, key, and value matrices, and then divide and reconstruct the feature map into multiple independent spatial windows, and calculate the attention weights in each window. The advantage of this method is that it can finely capture local features in a smaller area while retaining the spatial dependencies between different windows. In addition, concatenating the results of multiple attention heads and then performing linear projection helps to fuse the information between different windows, thereby enhancing the ability to express the spatial features of the image.

[0075] Reference Manual Attached Figure 5 , showing a flow chart of a channel self-attention mechanism provided by an embodiment of the present invention.

[0076] Figure 5 In , Channel-Wise Self-Attention represents the channel self-attention mechanism. V C represents the reshaped value matrix, Q C represents the reshaped query matrix, T represents transpose, K C Represents the reshaped key matrix. The Softmax operation is used to apply the Softmax function to the attention weight matrix to normalize it into a probability distribution. The matrix size is C×C, which represents the similarity between each channel. Y C The feature representing the attention of the channel window, the feature after the channel self-attention mechanism, Y C The size is H×W×C.

[0077] For channel self-attention, the input reshaped value matrix first generates the reshaped query matrix and the reshaped key matrix, and transposes the reshaped key matrix. Through matrix multiplication, the dot product of the reshaped query matrix and the reshaped key matrix is ​​calculated to generate the similarity matrix (attention matrix) between channels. The similarity matrix is ​​normalized by Softmax to generate the weight relationship matrix between each channel. The input reshaped value matrix is ​​weighted summed to obtain the feature representation after attention. Y C , and finally reshape the feature map into the same H×W×C dimension as the input feature for subsequent processing.

[0078] In a possible implementation, S302 specifically includes:

[0079] S3021: Generate the query matrix, key matrix and value matrix of the shallow features of the image to be processed according to the channel window self-attention of the channel dimension:

[0080]

[0081] in, Q represents the query matrix, K represents the key matrix, V represents the value matrix, X Represents shallow features, W Q , W K and W V Represents a linear projection.

[0082] S3022: Reshape the query matrix, key matrix, and value matrix to the same size .

[0083] in, H represents the height of the query matrix, key matrix, and value matrix, W represents the width of the query matrix, key matrix, and value matrix, C Represents the number of channels for the query matrix, key matrix, and value matrix inputs.

[0084] S3023: Separate the reshaped query matrix, key matrix and value matrix into h Size:

[0085]

[0086] in, Q c represents the reshaped query matrix, K c represents the reshaped key matrix, V c represents the reshaped value matrix, Indicates i The channel self-attention output is Indicates i The reshaped value matrix in the attention head, Indicates i The reshaped query matrix in the attention head, T represents transpose, Indicates i The reshaped key matrix in the attention head, represents a learnable temperature parameter.

[0087] S3024: Output of each attention head To splice:

[0088]

[0089] in, Y c Features representing channel window attention.

[0090] It should be noted that through the self-attention mechanism in the channel dimension, the model can effectively capture and enhance the inter-channel dependencies of image features. This mechanism generates query, key, and value matrices, reshapes them into appropriate sizes, and then performs multi-head attention calculations, so that the model can process each channel independently, thereby finely mining the relationship between channels in the feature map. At the same time, through the parallel calculation and splicing of multiple attention heads, the model can not only improve the expressiveness of features, but also focus on important features of different channels.

[0091] Reference Manual Attached Figure 6 , shows an image processing flow based on multi-scale feature extraction and convolution operation provided by an embodiment of the present invention.

[0092] Figure 6In the figure, AdaptiveMaxPool with a Kernel Sizing Factor represents an adaptive maximum pooling operation, which can adjust the size of the pooling window according to different downsampling scales (2, 4, 8). DW Conv3x3 represents the convolution kernel size of the depthwise separable convolution is 3x3. Nearest Upsampling represents the nearest neighbor upsampling. Concatenation represents the feature concatenation operation, which is used to concatenate feature maps of different scales in the channel dimension. Conv3x3 represents the convolution kernel size of the convolution layer is 3x3. Element-wise Product represents the element-wise product operation, which is used to multiply the fused features with the input features element by element to enhance key details. GELU Activation represents the GELU activation function, which provides a smooth nonlinear activation effect.

[0093] Specifically, multi-scale image information is captured by downsampling at different scales, and then the multi-scale information is fused through convolution, upsampling and splicing operations. Finally, the fused features are element-wise multiplied with the input features, and then nonlinearly processed through the activation function to generate an enhanced output feature map. This multi-scale feature extraction and fusion method is very effective in tasks such as image restoration and super-resolution reconstruction.

[0094] In a possible implementation manner, S303 specifically includes:

[0095] S3031: extracting local features and global features through convolution operation and global feature extraction operation respectively.

[0096] Among them, the convolution operation is a basic operation in the convolutional neural network (CNN), which is used to extract local features in the image. The convolution operation performs a dot product operation on the input image by sliding a convolution kernel (or filter) to identify local information such as edges, textures, corners, etc. in the image. The global feature extraction operation is an operation specifically used to extract global information from the entire image. Common global feature extraction methods include global average pooling, global maximum pooling, global convolution, attention mechanism, etc. Global features refer to features extracted from the entire image, which contain global information or contextual information of the image. Unlike local features, global features emphasize the overall semantics, structure or pattern of the entire image.

[0097] S3032: Fuse local features and global features to obtain comprehensive features:

[0098]

[0099] in,X local Represents local features, Represents global features, Conv 3x3 means the convolution kernel is 3x3 in size. GlobalPooling represents global pooling, Represents comprehensive characteristics, represents a learnable parameter for balancing local and global features.

[0100] S3033: Calculate query token based on comprehensive features:

[0101]

[0102] in, Q X represents a query token, D Represents a dictionary, K D represents a key dictionary, V D Represents a dictionary of values, , , , d represents the dimension of the input feature vector, r Indicates the reduction ratio.

[0103] S3034: Enhance query tagging through cross-attention calculation operations based on key dictionary and value dictionary:

[0104]

[0105] in, S represents the similarity matrix between the query token and the key dictionary, A represents the attention weight matrix, It means calculating the cosine similarity between the query token and the key dictionary token. represents a learnable parameter used to adjust the range of similarity values, TCGA represents the token dictionary cross-global attention operation, Y D Features representing global attention based on token dictionary intersection.

[0106] It should be noted that by combining local features and global features, the model can capture the detailed information and overall structure of the image at the same time. Specifically, the extraction of local features helps to retain fine parts such as edges and textures in the image, while the extraction of global features can identify context and global dependencies in a larger range. In addition, the fusion of local features and global features enables the model to not only focus on local details during the image restoration process, but also understand the overall semantics, thereby achieving better image reconstruction effects.

[0107] In a possible implementation, S3 processes shallow features through a feature aggregation module to extract deep features, specifically including:

[0108] S304: Segment the shallow features of the image to be processed, perform deep convolution on a portion of the shallow feature sub-feature maps obtained by segmentation, and send the rest to the multi-scale feature generation module:

[0109]

[0110] in, Split represents the split operation, X 0. X 1. X 2 and X 3 means pass Split The four shallow feature sub-feature maps segmented by the operation are represents the output feature map after deep convolution processing, DW-Conv 3x3 represents the depthwise convolution operation, up p Indicates that the feature is upsampled to the original resolution by nearest neighbor interpolation p , i Represents the feature map index variable used to identify different scales, Indicates downsampling the features to .

[0111] S305: Perform a multi-scale feature merging operation based on the multi-scale features obtained by nearest neighbor interpolation and adaptive maximum pooling, and Convolution to integrate local and global information:

[0112]

[0113] in, represents the output deep feature map, Indicates that the convolution kernel is a 1x1 convolution. Represents a concatenation operation.

[0114] S306: Normalize the output deep feature map through the Gaussian error linear unit, and adaptively adjust the shallow features of the image to be processed according to the attention map:

[0115]

[0116] in, Represents the final output deep feature map, represents the GELU function, Represents an element-wise multiplication operation.

[0117] S307: The extracted final output deep feature map is passed to the convolution channel feature mixer (CCFM) for channel mixing:

[0118]

[0119] in, Y Represents deep features.

[0120] Reference Manual Attached Figure 7 , showing the structure of a convolutional neural network module provided by the present invention.

[0121] Figure 7 In the figure, Conv 3x3 represents a 3x3 convolution layer, which is usually used to extract local features of the image. Conv 1x1 represents a 1x1 convolution layer. The main function of 1x1 convolution is to adjust the channel and change the depth of the feature map. At the same time, some linear combinations can be introduced to enhance the expressiveness of the features. The circle usually represents the additive fusion of the results of the two convolution operations.

[0122] S4: Fuse the attention features, deep features and shallow features to obtain fused features.

[0123] Among them, fusion features refer to the merging or integration of multiple features (such as shallow, deep and attention features) so that the model can simultaneously utilize information from different levels.

[0124] It should be noted that by fusing shallow, deep and attention features, the model can simultaneously utilize information at multiple different levels, thereby improving the completeness and richness of feature expression. This multi-level fusion method can more effectively deal with complex degradation situations in images and improve the quality and detail fidelity of the final image restoration.

[0125] In a possible implementation, S4 specifically includes:

[0126] S401: Fusing the attention feature, the deep feature and the shallow feature to obtain a fused feature;

[0127]

[0128] in, F D represents the deep features obtained by fusion, Represents an addition operation.

[0129] S402: Perform image reconstruction based on the fusion features and output the restored image:

[0130]

[0131] in, I HR represents the restored image, H RC Represents a reconstruction module consisting of a 3x3 convolutional layer and a pixel shuffle convolutional layer.

[0132] It should be noted that by fusing shallow features, deep features and attention features, effective integration of multi-dimensional and multi-level information is achieved, enabling the model to capture the rich details and global semantics of the image as a whole. This feature fusion method can retain the key information of the original image to the greatest extent while avoiding information loss and improving the image restoration effect. In addition, image reconstruction is performed in combination with the reconstruction module to ensure the high quality of the output image, so that the restored image has significant improvements in clarity, detail fidelity and global consistency.

[0133] S5: Perform image restoration on the image to be processed according to the fusion features, and output the restored image.

[0134] Among them, the restored image is processed, feature-fused and reconstructed to obtain the final output image, which should have significant improvements in details, clarity, contrast, etc.

[0135] It should be noted that by using the fused features for image reconstruction, it can be ensured that the model makes full use of multi-level information from shallow layers, deep layers and attention mechanisms. This can not only preserve the details of the image, but also enhance the global consistency of the image and improve the overall quality of the image. At the same time, using the fused features for image reconstruction can better eliminate the degradation effects in the image, such as blur, noise and low resolution problems, thereby obtaining a restored image with better visual effects and richer details.

[0136] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0137] In the present invention, by acquiring the image to be processed and gradually extracting shallow and deep features, combined with the triple attention module and the feature aggregation module, the image features can be comprehensively enhanced from multiple dimensions and levels. The triple attention module accurately captures key details and optimizes feature representation through the mechanisms of space, channel and global attention, while the feature aggregation module effectively fuses shallow and deep features to achieve comprehensive utilization of multi-scale information. Finally, the reconstruction step after feature fusion ensures the high quality of the restored image, making the method more robust and accurate when processing complex degraded images. By introducing a multiple attention mechanism, the method can effectively fuse and select features between information of different scales and types, thereby improving the quality and generalization ability of image restoration, enhancing the clarity, resolution and usability of the image, and providing reliable data support for various image analysis and visual tasks, thereby meeting the user's demand for high-quality images and improving user satisfaction.

[0138] Reference Manual Attached Figure 8 , showing a structural schematic diagram of an image restoration system based on a multiple attention mechanism provided by the present invention.

[0139] The present invention further provides an image restoration system 20 based on a multiple attention mechanism, which is applied to the above-mentioned image restoration method based on a multiple attention mechanism, comprising:

[0140] Processor 201.

[0141] The memory 202 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 201, an image restoration method based on a multiple attention mechanism as in the method embodiment is implemented.

[0142] The image restoration system 20 based on the multiple attention mechanism provided by the present invention can execute the above-mentioned image restoration method based on the multiple attention mechanism and achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0143] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0144] In the present invention, by acquiring the image to be processed and gradually extracting shallow and deep features, combined with the triple attention module and the feature aggregation module, the image features can be comprehensively enhanced from multiple dimensions and levels. The triple attention module accurately captures key details and optimizes feature representation through the mechanisms of space, channel and global attention, while the feature aggregation module effectively fuses shallow and deep features to achieve comprehensive utilization of multi-scale information. Finally, the reconstruction step after feature fusion ensures the high quality of the restored image, making the method more robust and accurate when processing complex degraded images. By introducing a multiple attention mechanism, the method can effectively fuse and select features between information of different scales and types, thereby improving the quality and generalization ability of image restoration, enhancing the clarity, resolution and usability of the image, and providing reliable data support for various image analysis and visual tasks, thereby meeting the user's demand for high-quality images and improving user satisfaction.

[0145] It should be understood that the processor in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0146] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0147] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0148] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0149] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0150] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0151] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0153] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0156] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0157] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, an image restoration method based on a multiple attention mechanism as in the method embodiment is implemented.

[0158] A computer-readable storage medium provided by the present invention can implement the steps and effects of the image restoration method based on the multiple attention mechanism of the above-mentioned method embodiment. To avoid repetition, the present invention will not go into details.

[0159] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0160] In the present invention, by acquiring the image to be processed and gradually extracting shallow and deep features, combined with the triple attention module and the feature aggregation module, the image features can be comprehensively enhanced from multiple dimensions and levels. The triple attention module accurately captures key details and optimizes feature representation through the mechanisms of space, channel and global attention, while the feature aggregation module effectively fuses shallow and deep features to achieve comprehensive utilization of multi-scale information. Finally, the reconstruction step after feature fusion ensures the high quality of the restored image, making the method more robust and accurate when processing complex degraded images. By introducing a multiple attention mechanism, the method can effectively fuse and select features between information of different scales and types, thereby improving the quality and generalization ability of image restoration, enhancing the clarity, resolution and usability of the image, and providing reliable data support for various image analysis and visual tasks, thereby meeting the user's demand for high-quality images and improving user satisfaction.

[0161] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0162] There are a few points to note:

[0163] (1) The drawings of the embodiments of the present invention only involve structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0164] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0165] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0166] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image restoration method based on a multiple attention mechanism, characterized in that: include: S1: Get the image to be processed; S2: extracting shallow features of the image to be processed through a convolutional layer; S3: Calculate the attention features through the triple attention module according to the shallow features; process the shallow features through the feature aggregation module to extract the deep features; S4: Fusing the attention feature, the deep feature and the shallow feature to obtain a fused feature; S5: performing image restoration on the image to be processed according to the fusion feature, and outputting the restored image; Among them, in S3, according to the shallow features, the attention is calculated through the triple attention module, specifically including: S301: Calculate the self-attention features of the spatial window based on the spatial dimension; S302: Calculate the self-attention feature of the channel window based on the channel dimension; S303: Calculate global attention features based on token dictionary intersection; Wherein, the S303 specifically includes: S3031: extracting local features and global features through convolution operations and global feature extraction operations respectively; S3032: Fusing the local features and the global features to obtain comprehensive features: in, Represents local features, Represents global features, Indicates that the convolution kernel is 3x3 in size. represents global pooling, Represents comprehensive characteristics, represents a learnable parameter for balancing local and global features; S3033: Calculate the query token according to the comprehensive features: in, represents a query token, D Represents a dictionary, represents a key dictionary, Represents a dictionary of values, represents the dimension of the input feature vector, Indicates the reduction ratio; S3034: According to the key dictionary and the value dictionary, the query tag is enhanced by a cross-attention calculation operation: in, represents the similarity matrix between the query token and the key dictionary, represents the attention weight matrix, It means calculating the cosine similarity between the query token and the key dictionary token. represents a learnable parameter used to adjust the range of similarity values, represents the token dictionary cross-global attention operation, Features representing global attention based on token dictionary intersection.

2. The image restoration method based on the multiple attention mechanism according to claim 1, characterized in that: The S2 is specifically: Through the convolution layer, the shallow features of the image to be processed are extracted according to the following formula: in, Represents the shallow features of the image to be processed, represents the image to be processed, Indicates that the convolution kernel size of the convolution layer is 3x3.

3. The image restoration method based on the multiple attention mechanism according to claim 1, characterized in that: The S301 specifically includes: S3011: Generate a query matrix, a key matrix and a value matrix of shallow features of the image to be processed according to the spatial window self-attention of the spatial dimension: in, represents the query matrix, represents the key matrix, represents the value matrix, Represents shallow features, represents a linear projection; S3012: Divide the query matrix, the key matrix, and the value matrix into non-overlapping windows respectively, and flatten the windows; S3013: According to the flattened window, the query matrix, the key matrix and the value matrix are respectively divided into Size: in, denote the projection query matrix, projection key matrix and projection value matrix respectively, Indicates i The query matrix in the attention head, Indicates i The key matrix in the attention head, Indicates i The value matrix in the attention head, represents the total number of attention heads, and the size of each attention head is , c represents the dimension of the channel, Indicates i The output of an attention head is represents the activation function, represents transpose, Relative position encoding; S3014: Concatenate the outputs of each attention head: in, Represents the characteristics of spatial window self-attention, Represents a splicing operation, represents the total number of attention heads, represents the spatial window self-attention operation, Represents the linear projection that fuses all features.

4. The image restoration method based on the multiple attention mechanism according to claim 1, characterized in that: The S302 specifically includes: S3021: Generate a query matrix, a key matrix, and a value matrix of shallow features of the image to be processed according to the channel window self-attention of the channel dimension: in, represents the query matrix, represents the key matrix, represents the value matrix, Represents shallow features, represents a linear projection; S3022: Reshape the query matrix, the key matrix, and the value matrix to have sizes in, represents the height of the query matrix, key matrix, and value matrix, represents the width of the query matrix, key matrix, and value matrix, Indicates the number of channels for query matrix, key matrix, and value matrix inputs; S3023: Separate the reshaped query matrix, key matrix and value matrix into Size: in, represents the reshaped query matrix, represents the reshaped key matrix, represents the reshaped value matrix, Indicates i The channel self-attention output is Indicates i The reshaped value matrix in the attention head, Indicates i The reshaped query matrix in the attention head, represents transpose, Indicates i The reshaped key matrix in the attention head, represents the learnable temperature parameter; S3024: Output of each attention head To splice: in, Features representing channel window attention.

5. The image restoration method based on multiple attention mechanism according to claim 1, characterized in that: In S3, the shallow features are processed by the feature aggregation module to extract the deep features, which specifically include: S304: Segment the shallow features of the image to be processed, and perform deep convolution on a portion of the shallow feature sub-feature graphs obtained by segmentation, and send the rest to a multi-scale feature generation module: in, Represents a split operation, Indicates that Operate the 4 shallow feature sub-feature maps segmented out, represents the output feature map after deep convolution processing, represents the depthwise convolution operation, Indicates that the feature is upsampled to the original resolution by nearest neighbor interpolation Represents the feature map index variable used to identify different scales, Indicates downsampling the features to size; S305: Perform a multi-scale feature merging operation based on the multi-scale features obtained by the nearest neighbor interpolation and adaptive maximum pooling, and Convolution to integrate local and global information: in, represents the output deep feature map, Represents the convolution kernel as Convolution of size, Represents a splicing operation; S306: normalizing the output deep feature map through a Gaussian error linear unit, and adaptively adjusting the shallow features of the image to be processed according to the attention map: in, Represents the final output deep feature map, express function, Represents an element-by-element multiplication operation; S307: The final output deep feature map extracted by the spatial segmentation feature module is passed to the convolution channel feature mixer (CCFM) for channel mixing: in, Represents deep features.

6. The image restoration method based on multiple attention mechanism according to claim 1, characterized in that: The S4 specifically includes: S401: Fusing the attention feature, the deep feature and the shallow feature to obtain a fused feature: in, represents the deep features obtained by fusion, Represents addition operation; S402: Perform image reconstruction according to the fusion features and output a restored image: in, represents the restored image, Indicated by The reconstruction module consists of a convolutional layer and a pixel shuffling convolutional layer.

7. An image restoration system based on a multi-attention mechanism, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the image restoration method based on the multiple attention mechanism as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the image restoration method based on a multiple attention mechanism as described in any one of claims 1 to 6.

Images