Efficient image restoration method and model based on non-local information mandatory modeling

Abstract

This application proposes an efficient image restoration method and model based on non-local information mandatory modeling, which relates to the field of image restoration technology. The method includes: using a convolutional layer to extract features from the image to be restored to obtain initial shallow features; using the initial shallow features as initial input parameters, performing n first iterative processes to obtain a first restored feature map; the first iterative process includes: performing at least one image restoration process on the input parameters and downsampling; using the results of multiple image restoration processes on the first restored feature map as input parameters, performing n second iterative processes to obtain a second restored feature map; the second iterative process includes: upsampling the input parameters and performing at least one image restoration process; and adding the results of processing the second restored feature map using a convolutional layer to the image to be restored to obtain a restored image. The above method can achieve image restoration of the image to be restored.

Description

Technical Field

[0001] The present application relates to the field of image restoration technology, and in particular to an efficient image restoration method and model based on mandatory modeling of non-local information. Background Art

[0002] In recent years, deep learning-based methods have been widely used in image restoration tasks and have gradually become the mainstream solution.

[0003] Convolutional neural networks (CNNs) have achieved remarkable results in image restoration, outperforming traditional restoration methods. However, the convolution operation of CNNs has static weights and a local receptive field, which to some extent limits their ability to remove non-local degenerate perturbations. Summary of the Invention

[0004] In view of this, this application proposes an efficient image restoration method and model based on mandatory modeling of non-local information.

[0005] In a first aspect, the present application provides an efficient image restoration method based on mandatory modeling of non-local information, comprising:

[0006] Use the convolution layer to extract features of the image to be restored and obtain the initial shallow features;

[0007] Using the initial shallow features as initial input parameters, performing n first iterative processes to obtain a first restored feature map; the first iterative process includes: performing at least one image restoration process on the input parameters, and downsampling the output features after the image restoration process;

[0008] The results of performing the image restoration process on the first restored feature map multiple times are used as input parameters, and a second iterative process is performed n times to obtain a second restored feature map; the second iterative process includes: upsampling the input parameters, and performing the image restoration process on the upsampled output features at least once; the image restoration process is used to achieve collaborative modeling of global features and local features;

[0009] The second restored feature map is processed using a convolution layer to generate a residual feature map, and the residual feature map is added to the image to be restored to obtain a repaired image.

[0010] In one embodiment, the image restoration processing includes: using hourglass attention to extract a target query vector from an input feature query, and using non-local top-k sampling on the input feature to obtain a target key and a target value, performing self-attention calculation on the target query vector, the target key, and the target value, and adding them to the input feature to output a first fused feature; normalizing the first fused feature and inputting it into a global feature compression branch and a deep convolution extraction branch respectively, fusing the outputs of the two branches and adding them to the first fused feature to generate a second fused feature.

[0011] In one embodiment, the efficient image restoration method based on non-local information mandatory modeling further includes:

[0012] After each image restoration process of the first iterative process is completed, the intermediate features output at the current resolution after the image restoration process is completed are retained and used as skip connection features;

[0013] In the second iterative processing, before each image restoration process, the current upsampling result is fused with the skip connection features of the corresponding scale.

[0014] In one embodiment, extracting a target query vector from an input feature query using hourglass attention, and obtaining a target key and a target value from the input feature using non-local top-k sampling, includes:

[0015] Normalize the input features and divide the window;

[0016] Apply convolutional layers to each divided window to increase the channel, split the channel to generate the original query vector, original key and original value;

[0017] Using hourglass attention to extract non-local dependencies from the original query vector to generate the target query vector;

[0018] Perform non-local top-k sampling on the original key and the original value to obtain the target key and the target value.

[0019] In one embodiment, extracting non-local dependencies from the original query vector using hourglass attention to generate the target query vector includes:

[0020] Performing a fully connected transformation on the original query vector using a first fully connected layer to map the original query vector to a low-dimensional superpixel space;

[0021] Use the GELU activation function to filter the negative features of the original query vector after the full connection transformation to obtain the first eigenvector;

[0022] Restoring the first eigenvector to the original space through a second fully connected layer to obtain a second eigenvector;

[0023] Use the Sigmoid function to process the second eigenvector to generate weights;

[0024] The weight is multiplied element-by-element by the original query vector and layer normalization is performed to obtain the target query vector.

[0025] In one embodiment, performing self-attention calculation on the target query vector, the target key, and the target value and adding them to the input feature to output a first fused feature includes:

[0026] Performing self-attention calculation on the target query vector, the target key, and the target value of each window to obtain an attention feature;

[0027] After the attention features are subjected to 1×1 convolution, the windows corresponding to the attention features are spliced ​​and restored to their original sizes to obtain window fusion features, where the original size is the size of the image to be restored;

[0028] The window fusion feature is added to the input feature, and the first fusion feature is output.

[0029] In one embodiment, normalizing the first fused feature and inputting it into a global feature compression branch and a depth convolution extraction branch respectively, fusing the outputs of the two branches and adding them to the first fused feature to generate a second fused feature, includes:

[0030] Processing the normalized first fusion feature through the standard FFN network of the global feature compression branch to output a first branch feature;

[0031] Processing the normalized first fusion feature through the FNN network of the deep convolution extraction branch to output a second branch feature, wherein the FNN network includes two fully connected layers and a 3×3 depthwise separable convolution layer located between the two fully connected layers;

[0032] The first branch feature and the second branch feature are fused and added to the first fused feature to generate the second fused feature.

[0033] In a second aspect, an embodiment of the present application further provides an efficient image restoration model based on non-local information mandatory modeling, wherein the efficient image restoration model based on non-local information mandatory modeling includes a first convolutional layer, a second convolutional layer, n cascaded first iterative processing modules, n cascaded second iterative processing modules, and a bottleneck layer;

[0034] The first convolution layer performs feature extraction on the image to be restored, obtains initial shallow features, and outputs the initial shallow features to the first iterative processing module of the first level;

[0035] The first iterative processing module includes a downsampling layer and at least one image complex atom module cascaded in sequence; the image complex atom module is used to combine the non-local attention mechanism with the local compensation FNN network to achieve collaborative modeling processing of global features and local features;

[0036] The bottleneck layer includes at least one image complex atom module, the input end of the bottleneck layer is connected to the output end of the first iterative processing module of the nth level, and the output end of the bottleneck layer is connected to the input end of the second iterative processing module of the first level;

[0037] The second iterative processing module includes an upsampling layer and at least one image complex atom module cascaded in sequence;

[0038] The second convolutional layer is used to receive the output features of the second iterative processing module of the nth level, and process the output features to generate a residual feature map, and add the residual feature map to the image to be restored to obtain a repaired image.

[0039] In one embodiment, the image complex atom module is used to use hourglass attention to extract a target query vector from an input feature query, and use non-local top-k sampling on the input feature to obtain a target key and a target value, and perform self-attention calculation on the target query vector, the target key, and the target value, and then add them to the input feature to output a first fusion feature; the first fusion feature is normalized and input into a global feature compression branch and a deep convolution extraction branch respectively, and the outputs of the two branches are fused and added to the first fusion feature to generate a second fusion feature.

[0040] In a third aspect, the present application also provides an efficient image restoration method based on mandatory modeling of non-local information, including:

[0041] Obtain the image to be restored;

[0042] The image to be restored is input into the efficient image restoration model based on non-local information mandatory modeling as described in the first aspect, and the image output by the efficient image restoration model based on non-local information mandatory modeling is determined as the repaired image of the image to be restored.

[0043] The present invention provides an efficient image restoration method based on mandatory modeling of non-local information, which has the following advantages over related technologies:

[0044] In this application, the image restoration process can perform deep processing on the input features to enhance the modeling capability of global information. Downsampling can halve the spatial size of the input features and double the number of channels, while upsampling can double the spatial size of the input features and halve the number of channels. On this basis, by taking the initial shallow features as the initial input parameters, performing n first iterations, and then taking the results of multiple image restoration processes on the first restored feature map as the input parameters, and performing n second iterations, it is possible to ensure the effective transmission and restoration of features in multi-scale space, while enhancing the modeling capability of the model for non-local information, removing non-local degenerate disturbances, better preventing information loss, preserving the information of the original defective image, and making the repaired complement image clearer and more complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 Schematic diagram of a flow chart of an efficient image restoration method based on mandatory modeling of non-local information in one embodiment of the present application;

[0047] Figure 2 This is a schematic diagram of a process of extracting a target query vector from an input feature query using hourglass attention in one embodiment of the present application;

[0048] Figure 3 This is a flowchart of non-local Top-K sampling in one embodiment of the present application;

[0049] Figure 4 Schematic diagram of the association between the global feature compression branch and the depth convolution extraction branch in one embodiment of the present application;

[0050] Figure 5 Schematic diagram of the structure of an efficient image restoration model based on mandatory modeling of non-local information in one embodiment of the present application;

[0051] Figure 6 Schematic diagram of the flow of an efficient image restoration method based on mandatory modeling of non-local information in another embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] In some embodiments, as Figure 1 As shown, Figure 1 As shown, an embodiment of the present application provides an efficient image restoration method based on non-local information mandatory modeling, and the efficient image restoration method based on non-local information mandatory modeling includes the following steps S101 to S104.

[0054] S101: Use the convolution layer to extract features of the image to be restored and obtain initial shallow features.

[0055] The size of the image to be restored can be 3×H×W, where 3 is the number of channels, H is the height of the image, and W is the width of the image. In step S101, a 3×3 convolutional layer can be used to extract initial shallow features, thereby helping to extract basic features while reducing the image size.

[0056] S102: Using the initial shallow features as initial input parameters, perform n first iterations to obtain a first restored feature map. The first iteration includes: performing at least one image restoration process on the input parameters, and downsampling the output features after the image restoration process. The image restoration process is used to achieve collaborative modeling of global features and local features. In one example, the image restoration process may include: extracting a target query vector from the input feature query using hourglass attention, obtaining a target key and a target value using non-local top-k sampling on the input features, performing self-attention calculation on the target query vector, the target key, and the target value, and then adding them to the input features to output a first fused feature; normalizing the first fused feature and inputting it into the global feature compression branch and the deep convolution extraction branch respectively, fusing the outputs of the two branches and adding them to the first fused feature to generate a second fused feature. The result of the i-th first iteration is used as the input of the i+1-th first iteration, and the number of executions of the image restoration process in the i-th first iteration is less than the number of executions of the image restoration process in the i+1-th first iteration. It should be noted that i and n are positive integers, and i is less than n.

[0057] It can be understood that in the image restoration process, the hourglass attention is used to query the target query vector for the input features, which can extract the non-local dependencies of the image to be restored, which is conducive to improving the high-quality restoration capability of complex degraded images. The use of non-local top-k sampling of the input features to obtain the target key and target value can avoid the interaction between local pixels, and only select the optimal pixel points in non-overlapping small blocks for calculation. This sparse attention mechanism not only reduces the redundancy of self-attention calculation, but also reduces the consumption of computing resources, thereby having stronger computing efficiency and practical application value. The use of a dual-branch network ensures that local details are not ignored. By setting a global feature compression branch and a deep convolution extraction branch, it is possible to supplement the local information of the corresponding features while maintaining the advantages of non-local information. This dual-branch structure can not only effectively restore global structural information, but also retain rich details, so that the present application can achieve superior performance in various image restoration tasks such as image deblurring, deraining, and defogging.

[0058] On this basis, the first iterative process includes image restoration and downsampling. Image restoration further extracts and transfers image features, while downsampling helps reduce the size of the feature map while increasing feature depth. For example, n can be 2. The initial shallow features undergo N1 image restoration processes, followed by the first downsampling. The results of the first downsampling are used as input features, and the image restoration processes are performed N2 times in sequence. The output results are then downsampled a second time, completing the encoding phase.

[0059] S103: Using the results of multiple image restoration processes on the first restored feature map as input parameters, perform n second iterative processes to obtain a second restored feature map. The second iterative process includes: upsampling the input parameters and performing at least one image restoration process on the upsampled output features. The result of the i-th second iterative process serves as the input for the (i+1)-th second iterative process, and the number of image restoration processes performed in the i-th second iterative process is greater than the number of image restoration processes performed in the (i+1)-th second iterative process. In one example, the number of image restoration processes performed in the i-th first iterative process is the same as the number of image restoration processes performed in the (n-i+1)-th second iterative process.

[0060] It can be understood that before performing n second iterative processes, multiple image restoration processes on the first restored feature map can enhance the modeling capability of global information while reducing redundant features. This process typically includes multiple continuous transformation operations, enabling the features to further extract and fuse key information in the low-resolution space, ensuring effective transmission and restoration of features in multi-scale space. Performing n second iterative processes on the input parameters can achieve decoding and gradually extract deep features, gradually restoring the features to their original resolution (i.e., the resolution corresponding to the initial shallow features). For example, n can be 2, and the input features undergo N2 image restoration processes, followed by the first upsampling. The results of the first upsampling are used as input features, and then undergo N1 image restoration processes in sequence. The output results are then upsampled a second time to complete the decoding phase.

[0061] S104: Using a convolutional layer to process the second restored feature map to generate a residual feature map, and adding the residual feature map to the image to be restored to obtain a restored image.

[0062] It is understood that the type of convolutional layer used in step S104 is the same as the type of convolutional layer used in step S101. If step S101 uses a 3×3 convolutional layer, the convolutional layer used in S104 is also a 3×3 convolutional layer. The second restored feature map is processed using a 3×3 convolutional layer to generate a residual feature map, and the residual feature map is added to the image to be restored to obtain a restored image, thereby restoring the image to be restored.

[0063] The above-mentioned efficient image restoration method based on mandatory modeling of non-local information uses image restoration processing to deeply process input features to enhance the modeling ability of global information. Downsampling can halve the spatial size of the input features and double the number of channels, while upsampling can double the spatial size of the input features and halve the number of channels. On this basis, by using the initial shallow features as the initial input parameters, performing n first iterations, and then using the results of multiple image restoration processes on the first restored feature map as the input parameters, and performing n second iterations, it is possible to ensure the effective transmission and restoration of features in multi-scale space, while enhancing the model's modeling ability for non-local information, removing non-local degenerate disturbances, better preventing information loss, preserving the information of the original defective image, and making the repaired complement image clearer and more complete.

[0064] In some embodiments, the efficient image restoration method based on mandatory modeling of non-local information also includes: after each image restoration process of the first iterative processing is completed, the intermediate features output at the current resolution after the image restoration process is completed are retained, and the intermediate features are used as jump connection features; in the second iterative processing process, before each image restoration process, the current upsampling result is fused with the jump connection features of the corresponding scale.

[0065] In practice, a skip connection involves appending input data directly to the output of a network layer. This design allows information to flow more freely while preserving the details and semantics of the original input data. This facilitates information propagation to subsequent layers and avoids information loss. Skip connections are typically implemented through summation or concatenation operations.

[0066] The corresponding operations of the first iterative processing and the second iterative processing may result in information loss. By adding a skip connection, in the first iterative processing, the intermediate features after each image restoration process are used as skip connection features. In the second iterative processing, the current upsampling result is fused with the skip connection features of the corresponding scale, and the fusion method is element-by-element addition. By fusing the current upsampling result with the skip connection features of the corresponding scale, the multi-scale context information extracted in the encoding stage can be fully utilized to enhance the image reconstruction capability in the decoding stage. At the same time, more detailed information in the image to be restored can be retained, which is conducive to preventing information loss, preserving the information of the original defective image, and making the repaired supplementary image clearer and more complete.

[0067] In some embodiments, during the image restoration process, hourglass attention is used to extract the target query vector from the input feature query, and non-local top-k sampling is used to obtain the target key and target value of the input feature, including: normalizing the input feature and dividing the window; applying the convolution layer to each divided window to increase the channel, splitting the channel to generate the original query vector, original key and original value; using hourglass attention to extract non-local dependency from the original query vector to generate the target query vector; performing non-local top-k sampling on the original key and original value to obtain the target key and target value.

[0068] In order to obtain the largest possible non-local receptive field, we divide the window into multiple small blocks with a specific size of M×M, where M can be adjusted according to the needs to obtain different receptive fields. In this embodiment, we can take M=32. Then, 1x1 convolution is applied to the divided windows to increase the channel and split the channel to generate the initial original query vector ( ), original key( ) and the original value ( ).

[0069] When generating the original query vector ( ), original key( ) and the original value ( ), the hourglass attention is used to extract the non-local dependencies of the original query vector to generate the target query vector. Figure 2 As shown, after the window is divided into multiple small blocks, a maximum pooling operation is used to store the original key ( ) and the original value ( ) is downsampled to obtain and , with the goal of reducing computational effort and expanding the receptive field. This approach removes features with small values ​​and limits local interactions, ensuring that only non-local information is modeled. Furthermore, we select top-k non-local features to enhance the sparsity of self-attention, while reducing redundant computation and improving efficiency. The NLTK sampling method can be expressed as follows:

[0070]

[0071]

[0072] and Represents the enhanced version of the original key K and original value V after Top-K selection, that is, the target key and target value, where , N is the number of elements in the feature map, usually representing the total number of pixels in the processed image. k is the number of pixel features selected within each block, usually representing the number of key features selected in each window or block.

[0073] In some embodiments, hourglass attention is used to extract non-local dependencies from the original query vector to generate a target query vector, including: using a first fully connected layer to perform a fully connected transformation on the original query vector to map the original query vector to a low-dimensional superpixel space; using a GELU activation function to filter negative features of the fully connected transformed original query vector to obtain a first feature vector; using a second fully connected layer to restore the first feature vector to the original space to obtain a second feature vector; using a Sigmoid function to process the second feature vector to generate weights; multiplying the weights by the original query vector element by element and performing layer normalization to obtain a target query vector. For example, the process of using hourglass attention to extract non-local dependencies from the original query vector to generate a target query vector can be referred to. Figure 3 .

[0074] In applications, directly capturing non-local dependencies incurs high computational costs. Therefore, in this embodiment, a first fully connected layer is used to aggregate non-local features in a low-dimensional superpixel space to reduce computational complexity. An activation function is then used to filter out negative features while retaining useful non-local information. Finally, a second fully connected layer restores the aggregated non-local features to their original resolution, completing efficient modeling of non-local information. The relationship for generating the target query vector is as follows:

[0075]

[0076]

[0077] in, represents the original query vector, represents the query vector after processing by the hourglass attention (HA) mechanism, i.e., the target query vector, By querying The intermediate feature representation obtained after applying the hourglass attention (HA) mechanism and the fully connected layer processing, represents two fully connected layers, represents the GELU activation function, represents the Sigmoid function, Stands for element-wise multiplication.

[0078] In one embodiment, performing self-attention calculation on the target query vector, target key, and target value and then adding them to the input features to output a first fused feature includes the following steps: performing self-attention calculation on the target query vector, target key, and target value of each window to obtain an attention feature; performing 1×1 convolution on the attention feature, concatenating the windows corresponding to each attention feature to restore them to their original size to obtain a window fused feature; and adding the window fused feature to the input features to output the first fused feature. The original size is the size of the image to be restored.

[0079] The self-attention mechanism can capture the association between different positions within the feature. By performing similarity-weighted aggregation between the target query vector and the target key and value in the same window, it can capture the local dependency within the window and generate independent attention features for each window.

[0080] Next, to further adjust the feature expression, we perform a 1×1 convolution on the attention output. This convolution does not change the channel dimension and is mainly used for linear transformation and lightweight fusion of features. Subsequently, the attention features within each window are restored and spliced ​​according to their spatial position, and reconstructed into a feature map of the same size as the original input image (for example, k M×M windows are spliced ​​and restored to the size of the original input image), realizing the fusion of multi-window contextual information, thereby improving the model's ability to comprehensively model global and local features. Finally, the window fusion features are directly added to the original input features through residual connections, which not only retains the underlying detailed information but also injects semantic associations enhanced by self-attention. The final output is the first fused feature that combines local perception and global dependency. This design reduces complexity through windowed calculations and ensures information flow through residual connections, which is conducive to the output of high-resolution feature maps.

[0081] In one embodiment, self-attention calculation is performed on the target query vector, target key and target value of each window to obtain an attention feature, including: calculating the correlation between the target query vector and the target key through the self-attention mechanism to obtain a correlation feature; normalizing the correlation feature and weighting it with the target value to obtain the attention feature.

[0082] Among them, the calculation formula of attention feature can be:

[0083]

[0084] in, is a learnable scaling parameter used to adjust The dot product magnitude of . Indicates the meaning of taking the transposition. Due to the adoption of the multi-head attention strategy, we concatenate the outputs of all multi-head attentions and obtain the results through linear projection. It should be noted that, combined with the above embodiment, taking k M×M windows as an example, the computational complexity of the standard self-attention is reduced from the original Reduce to ,in , which greatly reduces the computational complexity of self-attention.

[0085] In one embodiment, the first fused feature is normalized and input into the global feature compression branch and the deep convolution extraction branch respectively, and the outputs of the two branches are fused and added to the first fused feature to generate a second fused feature, including: processing the normalized first fused feature through the standard FFN network of the global feature compression branch to output the first branch feature; processing the normalized first fused feature through the FNN network of the deep convolution extraction branch to output the second branch feature; fusing the first branch feature and the second branch feature and adding them to the first fused feature to generate the second fused feature. The FNN network includes two fully connected layers and a 3×3 depthwise separable convolution layer located between the two fully connected layers. Figure 4 As shown in the figure, the global feature compression branch and the deep convolution extraction branch are parallel branches. The global feature compression branch uses the standard FFN network to process the first fusion feature after normalization, and the deep convolution extraction branch uses the FNN network to process the first fusion feature after normalization.

[0086] Among them, is the first fusion feature of the input Perform layer normalization and then input them into two independent branches. Where C represents the number of channels, H and W represent the height and width of the first fusion feature respectively.

[0087] The global feature compression branch includes a standard FFN network, which performs channel information interaction through a two-layer fully connected network and a GELU activation function. In this embodiment, for example, the first fusion feature after layer normalization can be expanded to 4C through 1×1 convolution, that is, , to enhance the feature expression ability; then processed by GELU activation function, and then reduced the number of channels back to the original level by 1×1 convolution , thereby outputting the first branch feature.

[0088] The deep convolution extraction branch includes an FNN network, which supplements local information by adding a 3×3 depthwise separable convolution layer between two fully connected layers. For example, the first fused feature after layer normalization is channel-expanded by a 1×1 convolution, mapping the feature to a higher-dimensional space for more comprehensive feature transformation. At this time, the feature dimension is expanded from C to 4C, that is, ; On the expanded feature map, 3×3 depth-wise separable convolution is applied to enhance the modeling capability of local pixel relationships while reducing the amount of computation; the channel-expanded features are transformed nonlinearly by the GELU activation function to screen effective features and enhance the model's expressiveness; the result processed by the GELU activation function is then compressed again by 1×1 convolution to restore the number of channels to the original input level, and the second branch feature is output, i.e. , ensuring that the feature dimension remains consistent with the standard FFN branch for subsequent fusion.

[0089] After obtaining the first branch feature and the second branch feature, the first branch feature and the second branch feature are finally summed to obtain a feature containing non-local information and rich local information, which is used as the input for the next step. The summation formula of the first branch feature and the second branch feature is:

[0090]

[0091] in, represents a standard feedforward network, represents a locally compensated feed-forward network. Indicates input Perform layer normalization operation, z represents the feature output in this step, and z is the result of element-wise summation after processing by the standard FFN and local compensation FFN branches.

[0092] The embodiment of the present application also provides an efficient image restoration model based on mandatory modeling of non-local information, such as Figure 5 As shown in FIG, the efficient image restoration model based on non-local information mandatory modeling includes a first convolutional layer, a second convolutional layer, n cascaded first iterative processing modules, n cascaded second iterative processing modules and a bottleneck layer. It should be noted that, Figure 5 The ONLformer Block in is the image complex atomic module.

[0093] The first convolutional layer extracts features from the image to be restored, obtains initial shallow features, and outputs the initial shallow features to the first-level first iterative processing module.

[0094] The first iterative processing module includes a downsampling layer and at least one image complex atom module cascaded in sequence. The image complex atom module is used to combine the non-local attention mechanism with the local compensation FNN network to achieve collaborative modeling processing of global features and local features. Furthermore, the image complex atom module can be used to extract the target query vector from the input feature query using hourglass attention, and obtain the target key and target value by non-local top-k sampling of the input feature, and add the target query vector, target key and target value to the input feature after self-attention calculation to output the first fused feature; the first fused feature is normalized and input into the global feature compression branch and the deep convolution extraction branch respectively, and the outputs of the two branches are fused and added to the first fused feature to generate the second fused feature; the output of the first iterative processing module of the i-th level is used as the input of the first iterative processing module of the i+1th level, and the number of image complex atom modules in the first iterative processing module of the i-th level is less than the number of image complex atom modules in the first iterative processing module of the i+1th level.

[0095] The bottleneck layer includes at least one image complex atomic module, the input end of the bottleneck layer is connected to the output end of the nth level first iterative processing module, and the output end of the bottleneck layer is connected to the input end of the first level second iterative processing module.

[0096] The second iterative processing module includes an upsampling layer and at least one image complex atomic module cascaded in sequence, wherein the output of the i-th second iterative processing module serves as the input of the i+1-th second iterative processing module, and the number of image complex atomic modules in the i-th second iterative processing module is less than the number of image complex atomic modules in the i+1-th second iterative processing module. Exemplarily, the number of image complex atomic modules in the i-th first iterative processing module is equal to the number of image complex atomic modules in the n-i+1-th second iterative processing module.

[0097] The second convolutional layer is used to receive the output features of the second iterative processing module of the nth level, and process the output features to generate a residual feature map, and add the residual feature map to the image to be restored to obtain a repaired image.

[0098] It can be understood that the efficient image restoration model based on mandatory modeling of non-local information in this embodiment is a symmetric U-shaped network. For example, n is 2, the downsampling layer is a 3×3 convolution with a stride of 2, and the upsampling layer is a pixel shuffling layer. The downsampling layer halves the spatial size of the restored feature map and doubles the number of channels, while the upsampling layer doubles the spatial size of the restored feature map and halves the number of channels. For the input image to be restored, we first apply a 3×3 convolution to obtain the initial shallow features. Then, the encoding phase begins: the restored image passes through N1 ONLformer image complex atom modules, followed by the first downsampling (halving the image size and doubling the number of channels). Subsequently, the image passes through N2 image complex atom modules, followed by the second downsampling, completing the encoding phase. After multiple downsampling steps in the encoding phase, the bottleneck phase (corresponding to the bottleneck layer) begins. In this phase, an additional Nm image complex atom modules are used to deeply process the feature map, enhancing the modeling of global information while reducing redundant features. During the decoding phase, the input feature map to be restored is first upsampled (the image size is doubled and the number of channels is halved). It then passes through N2 image complex atom modules and is added to the restored feature map of the corresponding scale during the encoding phase. This is followed by a second upsampling phase, passing through N1 image complex atom modules and adding the corresponding feature map. The result of this addition is then added to the image to be restored, ultimately generating the restored image.

[0099] In the application, for example, the image complex atom module can be trained end-to-end without additional pre-training. The Adam optimizer (β1=0.9, β2=0.999, weight decay is 0) is used during training, and the initial learning rate is set to 2×10⁻ 4and gradually decays to 1×10⁻ through cosine annealing scheduling 6 During training, we apply horizontal flipping and rotation for data augmentation, and keep the original size of the image during testing. The network consists of a three-layer structure, and the number of self-attention heads is set to [1, 2, 4]. The training parameters can be set separately for different tasks. In the deraining task, Rain200H / L, DID-Data, DDN-Data (synthetic data), and SPA-Data (real data) can be used for training, with the input image cropped to 128×128, the batch size set to 8, and 2000 or 400 training rounds (depending on the dataset). In the dehazing task, the RESIDE dataset (ITS and OTS subsets) can be used, the image cropped to 256×256, the batch size set to 2, and the training rounds to 300 or 20 rounds. In the deblurring task, the GoPro, HIDE, and RealBlur datasets are used, the crop size is 256×256, the batch size is set to 1 or 2, and the training rounds are 2400. For the denoising task, you can use the DIV2K, Flickr2K, BSD500, and WED datasets, with a crop size of 128×128, a batch size of 8, training for 200 rounds, and testing on Set12, BSD68, and Urban100.

[0100] It should be noted that the efficient image restoration model based on non-local information mandatory modeling provided in the embodiment of the present application and the efficient image restoration method based on non-local information mandatory modeling provided in the above embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the efficient image restoration model based on non-local information mandatory modeling. The beneficial effects of the efficient image restoration model based on non-local information mandatory modeling in the embodiment of the present application relative to the relevant technology are the same as the beneficial effects of the efficient image restoration method based on non-local information mandatory modeling in the above embodiment relative to the relevant technology, and the repeated parts will not be repeated.

[0101] In one embodiment, after the first iterative processing module at level i completes feature extraction, the current output is saved as a skip connection feature, and the skip connection feature is used for subsequent input feature fusion in the second iterative processing module at level n-i+1; before the second iterative processing module at level n-i+1 performs image restoration processing, the input feature in the second iterative processing module at level n-i+1 is fused with the corresponding skip connection feature.

[0102] In some embodiments, as Figure 6 As shown, the present application also provides an efficient image restoration method based on mandatory modeling of non-local information, including the following steps S601 and S602.

[0103] S601: Acquire the image to be restored.

[0104] S602: Input the image to be restored into an efficient image restoration model based on mandatory non-local information modeling, and determine the image output by the efficient image restoration model based on mandatory non-local information modeling as the inpainted image of the image to be restored. The efficient image restoration model based on mandatory non-local information modeling in this embodiment is the efficient image restoration model based on mandatory non-local information modeling in any of the above-mentioned solutions.

[0105] It should be noted that the beneficial effects of the efficient image restoration method based on non-local information mandatory modeling in the embodiment of the present application relative to the relevant technology are the same as the beneficial effects of the efficient image restoration model based on non-local information mandatory modeling in the above-mentioned embodiment relative to the relevant technology, and the repeated parts will not be repeated.

[0106] In some embodiments, an electronic device provided by an embodiment of the present application includes a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the efficient image restoration method based on non-local information mandatory modeling of any of the above-mentioned schemes.

[0107] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for executing different actions of the method flow according to the embodiments of the present application.

[0108] Memory, for example, can be any medium capable of containing, storing, conveying, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, components, or propagation media. Specific examples of memory include: magnetic storage devices, such as magnetic tape or hard disk drives (HDDs); optical storage devices, such as compact discs (CD-ROMs); random access memory (RAM) or flash memory; and / or wired or wireless communication links.

[0109] This application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the efficient image restoration method based on non-local information mandatory modeling of any of the above-mentioned solutions. The computer-readable medium may be included in the device / apparatus / system described in the above-mentioned embodiments; or it may exist independently and not be incorporated into the device / apparatus / system. The computer-readable medium carries one or more programs, and when executed, implements the method of the embodiments of this application.

[0110] According to an embodiment of the present application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, radio frequency signals, or any suitable combination thereof.

[0111] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present application. Therefore, the scope of the present application should not be limited to the above-mentioned embodiments, but should be determined not only by the attached claims, but also by the equivalents of the attached claims. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An efficient image restoration method based on mandatory modeling of non-local information, characterized by: include: Use the convolution layer to extract features of the image to be restored and obtain the initial shallow features; The initial shallow feature is used as the initial input parameter, and the first iterative processing is performed n times to obtain the first restored feature map; the first iterative processing includes: performing at least one image restoration processing on the input parameter, and downsampling the output feature after the image restoration processing; the image restoration processing is used to realize the collaborative modeling processing of global features and local features; the image restoration processing includes: using hourglass attention to extract the target query vector from the input feature query, and using non-local top-k sampling to obtain the target key and target value from the input feature, performing self-attention calculation on the target query vector, the target key and the target value, and adding them to the input feature to output the first fusion feature. feature; normalizing the first fused feature and inputting it into the global feature compression branch and the deep convolution extraction branch respectively; processing the normalized first fused feature through the standard FFN network of the global feature compression branch to output a first branch feature; processing the normalized first fused feature through the FNN network of the deep convolution extraction branch to output a second branch feature; fusing the first branch feature and the second branch feature and adding them to the first fused feature to generate a second fused feature; wherein the FNN network of the deep convolution extraction branch includes two fully connected layers and a 3×3 depthwise separable convolution layer located between the two fully connected layers; Using the results of performing the image restoration process on the first restored feature map multiple times as input parameters, performing a second iterative process n times to obtain a second restored feature map; the second iterative process includes: upsampling the input parameters, and performing the image restoration process on the upsampled output features at least once; The second restored feature map is processed using a convolution layer to generate a residual feature map, and the residual feature map is added to the image to be restored to obtain a repaired image.

2. The efficient image restoration method based on non-local information mandatory modeling according to claim 1, characterized in that: The efficient image restoration method based on non-local information mandatory modeling also includes: After each image restoration process of the first iterative process is completed, the intermediate features output at the current resolution after the image restoration process is completed are retained and used as skip connection features; In the second iterative processing, before each image restoration process, the current upsampling result is fused with the skip connection features of the corresponding scale.

3. The efficient image restoration method based on non-local information mandatory modeling according to claim 1, characterized in that: The method of extracting a target query vector from an input feature query using hourglass attention and obtaining a target key and a target value from the input feature using non-local top-k sampling includes: Normalize the input features and divide the window; Apply convolutional layers to each divided window to increase the channel, split the channel to generate the original query vector, original key and original value; Using hourglass attention to extract non-local dependencies from the original query vector to generate the target query vector; Perform non-local top-k sampling on the original key and the original value to obtain the target key and the target value.

4. The efficient image restoration method based on non-local information mandatory modeling according to claim 3, characterized in that: The extracting non-local dependencies from the original query vector using hourglass attention to generate the target query vector includes: Performing a fully connected transformation on the original query vector using a first fully connected layer to map the original query vector to a low-dimensional superpixel space; Use the GELU activation function to filter the negative features of the original query vector after the full connection transformation to obtain the first eigenvector; Restoring the first eigenvector to the original space through a second fully connected layer to obtain a second eigenvector; Use the Sigmoid function to process the second eigenvector to generate weights; The weight is multiplied element-by-element by the original query vector and layer normalization is performed to obtain the target query vector.

5. The efficient image restoration method based on non-local information mandatory modeling according to claim 3, characterized in that: Performing self-attention calculation on the target query vector, the target key, and the target value, and then adding them to the input feature to output a first fused feature, including: Performing self-attention calculation on the target query vector, the target key, and the target value of each window to obtain an attention feature; After the attention features are subjected to 1×1 convolution, the windows corresponding to the attention features are spliced ​​and restored to their original sizes to obtain window fusion features, where the original size is the size of the image to be restored; The window fusion feature is added to the input feature, and the first fusion feature is output.

6. An efficient image restoration system based on mandatory modeling of non-local information, characterized by: The efficient image restoration system based on non-local information mandatory modeling includes a first convolutional layer, a second convolutional layer, n cascaded first iterative processing modules, n cascaded second iterative processing modules and a bottleneck layer; The first convolution layer performs feature extraction on the image to be restored, obtains initial shallow features, and outputs the initial shallow features to the first iterative processing module of the first level; The first iterative processing module includes a downsampling layer and at least one image complex atom module cascaded in sequence; the image complex atom module is used to combine the non-local attention mechanism and the local compensation FNN network to realize the collaborative modeling processing of global features and local features; the image complex atom module is used to extract the target query vector from the input feature query using hourglass attention, and obtain the target key and target value by non-local top-k sampling on the input feature, and add the target query vector, the target key and the target value to the input feature after self-attention calculation to output a first fusion feature; the first fusion feature is normalized and input into the global feature compression branch and the deep convolution extraction branch respectively, and the normalized first fusion feature is processed by the standard FFN network of the global feature compression branch to output a first branch feature; the normalized first fusion feature is processed by the local compensation FNN network of the deep convolution extraction branch to output a second branch feature; Fusing the first branch feature and the second branch feature and adding the first fused feature to generate a second fused feature; The bottleneck layer includes at least one image complex atom module, the input end of the bottleneck layer is connected to the output end of the first iterative processing module of the nth level, and the output end of the bottleneck layer is connected to the input end of the second iterative processing module of the first level; The second iterative processing module includes an upsampling layer and at least one image complex atom module cascaded in sequence; The second convolutional layer is used to receive the output features of the second iterative processing module of the nth level, and process the output features to generate a residual feature map, and add the residual feature map to the image to be restored to obtain a repaired image.

7. An efficient image restoration method based on non-local information mandatory modeling, characterized in that: include: Obtain the image to be restored; The image to be restored is input into the efficient image restoration system based on non-local information mandatory modeling as described in claim 6, and the image output by the efficient image restoration system based on non-local information mandatory modeling is determined as the repaired image of the image to be restored.

Images