Image restoration system and method based on dense sampling and adaptive sparse sampling

Abstract

The invention discloses an image restoration system and method based on dense sampling and adaptive sparse sampling, and the system comprises an image feature extraction module, a feature perception module, a Transform feature processing module based on dense sampling and adaptive sparse sampling, and an image reconstruction module. Meanwhile, the local feature information and the global feature information of the image are extracted, the relevance between the features is adjusted more effectively while the calculation efficiency is guaranteed, the high-quality image is reconstructed, and the model performance is remarkably improved.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, in particular to an image restoration system and method based on dense sampling and adaptive sparse sampling. Background Art

[0002] Existing image restoration methods based on the Transformer architecture generally use window-based dense sampling or fixed-interval-based sparse sampling to sample the query vector query, key vector key, and value vector value when designing the self-attention mechanism. Although these two methods can reduce the computational complexity, they do not take into account the intrinsic information representation of the features, and sparse sampling cannot adaptively adjust the sampling range to extract image features, so their improvement on model performance is limited. Summary of the invention

[0003] In order to solve the problem that the performance of the existing models is not significantly improved, the present invention proposes an image restoration system and method based on dense sampling and adaptive sparse sampling. The system and method can simultaneously extract the local feature information and global feature information of the image according to the internal representation information of the image, and more effectively adjust the correlation between features while ensuring the computational efficiency, so as to reconstruct a high-quality image and significantly improve the model performance.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to an image restoration system based on dense sampling and adaptive sparse sampling, comprising: an image feature extraction module, a feature perception module, a Transformer feature processing module based on dense sampling and adaptive sparse sampling, and an image reconstruction module, wherein: the image feature extraction module is based on the input low-quality image I LQ ∈R H×D×3 , perform image feature extraction and obtain the first feature F 0 ∈R H×D×C The feature perception module is based on the first feature F 0 , perform feature internal information perception and calculation processing to obtain the updated second feature F 1 ∈R H×D×C and the learned variable sparse coefficient I; the Transformer feature processing module is based on the second feature F 1 The variable sparse coefficient I is used to perform intra-feature token attention interaction processing to obtain the updated result feature F 2 ∈R H×D×C The image reconstruction module is based on the processed result feature F 2 , reconstruct high-quality image I HQ .

[0006] The Transformer feature processing module includes: K residual module groups (RG) and a mapping layer (Mapping), wherein: the residual module group receives the output result F of the feature perception module 1 , cyclic processing until the number of residual module groups that meet the processing reaches K, and the processed image features are obtained; the mapping layer performs a layer of 3×3 convolution processing according to the image features to obtain the mapped features F map ; Feature F after mapping map and the initial input feature F 1 After adding through the skip connection, the updated result feature F is obtained 2 .

[0007] The residual module group includes: a dense sampling Transformer submodule (DTB) with a total number of modules N, an adaptive sparse sampling Transformer submodule (sTB) and a mapping layer, wherein: the 2i-1th submodule is set as a dense sampling Transformer submodule, the 2ith submodule is set as an adaptive sparse sampling Transformer submodule, and the last processing unit is set as a mapping layer, wherein: the i-th submodule receives the output result of the i-1th submodule, and obtains a new image feature after internal feature processing, and thus loops until the number of Transformer submodules processing features reaches N, and the mapping layer performs a layer of 3×3 convolution processing according to the processed image features to obtain the mapped feature F m , and then the mapped feature F m and the input feature F of the first submodule in By adding the skip connection, we get the output feature F out .

[0008] The dense sampling Transformer submodule includes: a layer normalization (LayerNorm) unit, a dense sampling unit, a multi-head self-attention mechanism (MSA) unit and a forward feedback (FFN) unit, wherein: the layer normalization unit is based on the input x 1 , normalize the feature values ​​of each layer of the channel dimension of the image to obtain an unbiased result; the dense sampling unit densely samples the feature map distribution by dividing the window according to the layer normalization result to obtain the resampled sequence token; the multi-head self-attention mechanism unit extracts multiple groups of query, key, and value vectors according to the resampled sequence token to calculate the result x after token interaction 2 ; Then output the result x 2 Through the skip connection and input x 1 Add to get the new result x 3 ; New result x 3After layer normalization, and then through the forward feedback network, two linear transformations and ReLU nonlinear activation function processing are performed to obtain the output feature result x 4 ; Output result x 4 Through the skip connection and input x 3 Add to get the final result x 5 .

[0009] The adaptive sparse sampling Transformer submodule includes: a layer normalization (LayerNorm) unit, an adaptive sparse sampling unit, a multi-head self-attention mechanism (MSA) unit and a forward feedback (FFN) unit, wherein: the layer normalization unit is based on the input x 1 , normalize the feature values ​​of each channel dimension of the image layer to obtain an unbiased result; the adaptive sparse sampling unit extracts tokens every I steps on the feature map based on the layer normalization result and uses the sparse coefficient I output by the feature perception module, and performs sparse sampling on the feature map distribution to obtain a resampled sequence token; the multi-head self-attention mechanism unit extracts multiple sets of query, key, and value vectors based on the resampled sequence token to calculate the result x after token interaction 2 ; Then output the result x 2 Through the skip connection and input x 1 Add to get the new result x 3 ; New result x 3 After layer normalization, it is then input into the forward feedback network, and two linear transformations and ReLU nonlinear activation functions are performed to obtain the output feature result x 4 ; Output result x 4 Through the skip connection and input x 3 Add to get the final result x 5 .

[0010] The image reconstruction module includes: a preprocessing unit, an upsample unit (Upsample) and an output unit, wherein: the preprocessing unit is based on the feature F output by the feature processing module 2 , perform 3×3 convolution processing to obtain new features; the upsampling unit performs a 3×3 convolution processing and a shuffle processing according to the output result of the preprocessing unit to obtain the upsampled feature result; the output unit performs a 3×3 convolution processing according to the upsampled feature result based on the super-resolution task or denoising and compression artifact removal task to obtain the output high-quality image I HQ Or according to the feature F output by the feature processing module 2 As input, a 3×3 convolution is performed, and the processed result is connected to the input low-quality image I through a jump connection. LQAdd together to get the final high-quality image I HQ . Technical Effects

[0011] The present invention utilizes the intrinsic correlation between features to obtain effective information within the features, obtains sparse coefficients through feature perception, and in the feature processing stage, utilizes window-based dense sampling and sparse coefficient-based sparse sampling, and uses self-attention interaction units to process features, adaptively extracts local and global information of the image, and further improves the quality of the generated image in the final image reconstruction stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the system of the present invention;

[0013] Figure 2 Schematic diagram of feature perception module;

[0014] Figure 3 It is a schematic diagram of feature processing module;

[0015] Figure 4 Schematic diagram of the image reconstruction module. DETAILED DESCRIPTION

[0016] like Figure 1 As shown, an image restoration system based on dense sampling and adaptive sparse sampling involved in this embodiment includes: an image feature extraction module, a feature perception module, a Transformer feature processing module based on dense sampling and adaptive sparse sampling, and an image reconstruction module. The image restoration method includes:

[0017] The first step is to select a single low-quality image from the commonly used image restoration test dataset and input it into the image feature extraction module. Specifically, a convolutional neural network is used to extract the image feature from the input image I LQ ∈R H×D×3 The first feature F is extracted from 0 ∈R H×D×C , where: H is the length of the image, D is the width of the image, the default number of image channels is 3, and C is the number of new channels for extracting features;

[0018] The convolutional neural network has one layer and the convolution kernel size is 3×3.

[0019] The second step is to transform the first feature F 0 ∈R H×D×C Input feature perception module, after regularization processing, the corresponding query vector query∈R is extracted through convolutional neural network HD×C , key vector key∈R hw×C Sum value vector value∈R hw×3After that, the attention matrix attn∈R is calculated HD×hw , update the value vector value through the attention matrix to obtain new output features and sparse coefficients I, including:

[0020] 2.1) For the first feature F 0 Regularization normalizes the process to obtain intermediate features Then, query∈R is obtained by a convolutional neural network with a convolution kernel size of 1×1, stride of 1, and padding of 0. HD×C , after being processed by a convolutional neural network with a convolution kernel size of 4×4, stride of 4, and padding of 0, the key vector key∈R is obtained hw×C Sum value vector value∈R hw×3 , where: h = H / / s, w = D / / s, / / means downsampling by s times;

[0021] The downsampling multiple s is preferably a multiple of 4.

[0022] 2.2) Calculate the updated first feature Where: attention matrix attn = query × key T , where: attn∈R HD×hw , Softmax is the activation function, C is the number of channels of each head during self-attention interaction, and T is the transposition operation;

[0023] In this embodiment, the number of heads when calculating attention interactions is preferably 6.

[0024] 2.3) Using the attention matrix and the new activation function sigmoid, we get the activation value of the attention matrix. Using this information distribution and the threshold gate operation, we set the activation value less than 0.5 to be unimportant, that is, 0, and the activation value not less than 0.5 to 1. We count the number of 1s to get the activation rate and sparse coefficient, specifically: activation rate λ = sum(gate(sigmoid(attn)) / num(attn), sparse coefficient Among them: sum() is to sum the matrix, num() is to count the number of elements in the matrix, and vs is the preset sparse initial value.

[0025] The attention matrix attn∈R HD×hwThe information contained inside represents the degree of correlation between image pixels. After activation by the sigmoid function, the correlation between image pixel information is mapped to the range of 0-1. Therefore, the activation values ​​less than 0.5 are further filtered out through the threshold gate operation. Therefore, the image pixel information retained is the pixel with strong representation ability. By counting the percentage of these pixels in the total number of image pixels, the overall representation information of the image can be judged. The stronger the information correlation between pixels, the more pixels are activated. Therefore, the larger the calculated activation rate is, the smaller the sparsity coefficient is.

[0026] The sparse initial value can be adjusted for different data sets and is 4 by default.

[0027] 2.4) Calculate the second feature based on the intermediate feature and the updated first feature Among them: proj() is a linear layer neural network.

[0028] Step 3: The second feature F 1 After being processed by multiple residual module groups RG in sequence, the processed features pass through a mapping layer and undergo 3×3 convolution processing to obtain output features. The output features are connected to the input features F through a skip connection. 1 Add the result F 2 . Among them: each RG module includes N Transformer submodules, the 2i-1th submodule is set as a window-based dense sampling Transformer submodule, and the 2ith submodule is set as a sparse sampling Transformer submodule based on a variable coefficient I. The features processed by N Transformer submodules pass through a layer of 3×3 convolutional neural network, and are added to the input features of the first submodule through skip connections to obtain the output features of the RG module.

[0029] The window-based dense sampling Transformer submodule processing process includes:

[0030] 3.1) The input features are normalized by layers to obtain the intermediate features F s , and then the H×D feature map is densely sampled, and the feature tokens are grouped and sampled according to the W×W window size. The number of tokens in each group is W×W, which are distributed in the dense area of ​​a certain window range of the original feature map. The updated intermediate feature Each group of token features F t ∈R WW×C .in, That is F n By multiple F t Concatenate along the first dimension.

[0031] 3.2) Each group of token features F t After three linear layers, query, key and value are extracted, and the features after self-attention interaction are calculated. Among them, Softmax is the activation function, T is the transposition operation, and C is the number of channels of each head during self-attention interaction.

[0032] 3.3) The feature F after self-attention interaction a After a linear layer, it is connected to the intermediate feature F through a skip connection. s After addition, the output is sent to the Feedforward Network (FFN) to obtain the final output features.

[0033] The FFN network is composed of two layers of linear networks and a Relu activation unit, wherein the output features of the FFN network are added to the input features of the FFN through a jump connection to obtain the final output feature result.

[0034] The sparse sampling Transformer submodule processing process based on the variable coefficient I includes:

[0035] 3.a) The input features are normalized to obtain the intermediate features_F s Then, according to the sparse coefficient I obtained in the second step, the feature map of H×D is sampled in groups with an interval of I according to the sparse sampling method. The number of tokens in each group is Distributed in the sparse area of ​​the original feature map, the updated intermediate features are obtained The characteristics of each token group are in, m=I×I,that is, _F n By multiple _F t Concatenate along the first dimension.

[0036] 3.b) Each group of token features_F t After three linear layers, query, key and value are extracted, and the features after self-attention interaction are calculated. Among them, Softmax is the activation function, T is the transposition operation, and C is the number of channels of each head during self-attention interaction.

[0037] 3.c) The feature after self-attention interaction_F a After being processed by a linear layer, the intermediate features are added through skip connection and normalization and then output to the feed-forward network (FFN) to obtain the final output features.

[0038] The FFN network is composed of two layers of linear networks and a Relu activation unit, wherein the output features of the FFN network are added to the input features of the FFN through a jump connection to obtain the final output feature result.

[0039] The fourth step is to output the feature F in the third step 2 The input image reconstruction module reconstructs a high-quality image. Specifically, for the super-resolution task, the second feature F2 is processed by a 3×3 convolution layer through the preprocessing unit to obtain the output feature F * =Conv 3×3 (F 2 ); New output F * After the upsampling unit (Upsample), a layer of 3×3 convolution processing and shuffle processing are performed, and the result is further processed by the output unit and a layer of 3×3 convolution processing is performed to obtain a high-quality image: I HQ =Conv 3×3 (Upsample(F * )); For tasks such as denoising and decompression artifacts, the second feature F 2 After the output unit, a layer of 3×3 convolution is performed, and the result is connected to the input low-quality image I through a jump connection. LQ Add together to get a high-quality image: I HQ =Conv 3×3 (F 2 )+I LQ .

[0040] The upsampling is implemented using, but not limited to, the sub-pixel convolutional neural network described in Shi, Wenzhe, et al., "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network" (Proceedings of the IEEE conference on computer vision and pattern recognition. 2016).

[0041] After specific actual experiments, in terms of hardware, the GPU model NVIDIA Ge Force RTX 3090 24GB is used. In terms of software environment, the system version of the server is Ubuntu18.04.6LTS, and the CUDA version is 11.6. All relevant codes involved in the experiment are written in Python language, and its version is 3.8.11. The machine learning library used to build the neural network is provided by Pytorch. 1) Set the super-resolution multiple to 2, and use DIV2K and Flickr2K as training data, with a total of 3550 high-resolution images and 3550 low-resolution images that are twice sampled by bicubic linear interpolation. The test data is 5 public data sets Set5, Set14, B100, Urban100 and Manga109. 2) The preset parameters are: the number of channels for feature extraction C = 180, the number of residual module groups RG is 4, and the number of Transformer modules inside each RG is 3 DTB Modules and 3 STB modules, the number of heads during attention interaction is 6, based on window-based dense sampling, the window size is set to 8. 3) Experimental process, based on the training data, training for 500,000 steps, training using L1loss, the optimizer using Adam, the initial learning rate 2e-4, and then training for [250,000, 400,000, 450,000, 475,000] steps, the learning rate is reduced by half successively to obtain the trained model, and the indicators PSNR and SSIM are tested on 5 test sets. The obtained indicator results are shown in Table 1.

[0042] Table 1 index Set5 Set14 B100 Urban100 Manga109 PSNR 38.3856 34.3636 32.4870 33.7451 39.9291 SSIM 0.9620 0.9251 0.9036 0.9417 0.9799

[0043] The experimental environment used in the above experimental process is consistent with that used in the previous experiment, and the same training data and test data are used for the control experiment. The only difference between the network structure used in the control experiment and the network built using this technology is that the feature perception module is abandoned to extract information about the internal correlation of the features, and the variable sparse coefficient is not used for sparse sampling. The sampling range is fixed during sparse sampling, and the sampling interval I is set to 4. The experimental process is based on the same training data and trained for 500,000 steps under the same experimental parameter settings to obtain the trained model. The indicators PSNR and SSIM are tested on 5 test sets, and the obtained indicator results are shown in Table 2.

[0044] Table 2 index set5 Set14 B100 U100 M109 PSNR 38.3132 34.2498 32.4418 33.4685 39.7416 SSIM 0.9618 0.9244 0.9031 0.9398 0.9793

[0045] Comparing Table 1 and Table 2, it is found that after using the technical improvement involved in the present invention, the two commonly used performance indicators tested on the five public data sets of the super-resolution task of the image restoration model have been improved. The reason is that the present invention learns the internal correlation information of the feature through the feature perception module, obtains the sparse coefficient to guide the feature processing module to perform adaptive sparse sampling, and then learns more effective feature information to guide the feature reconstruction module to obtain a higher quality image. While improving performance, the present invention uses an adaptive sparse coefficient for sparse sampling, which can save more computing resources and improve computing efficiency compared to fixed interval sparse sampling.

[0046] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. An image restoration system based on dense sampling and adaptive sparse sampling, characterized in that: include: Image feature extraction module, feature perception module, Transformer feature processing module based on dense sampling and adaptive sparse sampling, and image reconstruction module, where: the image feature extraction module is based on the input low-quality image I LQ ∈R H×D×3 , perform image feature extraction and obtain the first feature F0∈R H×D×C The feature perception module perceives and calculates the internal information of the feature based on the first feature F0 to obtain the updated second feature F1∈R H×D×C and the learned variable sparse coefficient I; the Transformer feature processing module performs intra-feature token attention interaction processing based on the second feature F1 and the variable sparse coefficient I to obtain the updated result feature F2∈R H×D×C The image reconstruction module reconstructs the high-quality image I according to the processed result feature F2. HQ .

2. The image restoration system based on dense sampling and adaptive sparse sampling according to claim 1, characterized in that: The Transformer feature processing module includes: K residual module groups (RG) and a mapping layer (Mapping), wherein: the residual module group receives the output result F1 of the feature perception module, and processes it cyclically until the number of residual module groups that meet the processing reaches K, and obtains the processed image features; the mapping layer performs a layer of 3×3 convolution processing according to the image features to obtain the mapped features F map ; Feature F after mapping map After adding it to the initial input feature F1 through a skip connection, the updated result feature F2 is obtained.

3. The image restoration system based on dense sampling and adaptive sparse sampling according to claim 2, characterized in that: The residual module group includes: a dense sampling Transformer submodule (DTB) with a total number of modules N, an adaptive sparse sampling Transformer submodule (STB) and a mapping layer, wherein: the 2i-1th submodule is set as a dense sampling Transformer submodule, the 2ith submodule is set as an adaptive sparse sampling Transformer submodule, and the last processing unit is set as a mapping layer, wherein: the i-th submodule receives the output result of the i-1th submodule, and obtains a new image feature after internal feature processing, and thus loops until the number of Transformer submodules processing the feature reaches N, and the mapping layer performs a layer of 3×3 convolution processing according to the processed image feature to obtain the mapped feature F m , and then the mapped feature F m and the input feature F of the first submodule in By adding the skip connection, we get the output feature F out .

4. The image restoration system based on dense sampling and adaptive sparse sampling according to claim 3, characterized in that: The dense sampling Transformer submodule includes: a layer normalization (LayerNorm) unit, a dense sampling unit, a multi-head self-attention mechanism (MSA) unit and a forward feedback (FFN) unit, wherein: the layer normalization unit normalizes the channel dimension feature values ​​of each layer of the image according to the input x1 to obtain an unbiased result; the dense sampling unit densely samples the feature map distribution by dividing the window according to the layer normalization result to obtain a resampled sequence token; the multi-head self-attention mechanism unit extracts multiple groups of query, key, and value vectors according to the resampled sequence token to calculate the result x2 after token interaction; then the output result x2 is added to the input x1 through a jump connection to obtain a new result x3; the new result x3 is layer normalized, and then passes through the forward feedback network, undergoes two linear transformations and ReLU nonlinear activation function processing, and obtains the output feature result x4; the output result x4 is added to the input x3 through a jump connection to obtain the final result x5.

5. The image restoration system based on dense sampling and adaptive sparse sampling according to claim 3, characterized in that: The adaptive sparse sampling Transformer submodule includes: a layer normalization (LayerNorm) unit, an adaptive sparse sampling unit, a multi-head self-attention mechanism (MSA) unit and a forward feedback (FFN) unit, wherein: the layer normalization unit normalizes the channel dimension feature values ​​of each layer of the image according to the input x1 to obtain an unbiased result; the adaptive sparse sampling unit extracts tokens every I steps on the feature map based on the layer normalization result and uses the sparse coefficient I output by the feature perception module to sparsely sample the feature map distribution to obtain a resampled sequence token; the multi-head self-attention mechanism unit extracts multiple groups of query, key, and value vectors according to the resampled sequence token to calculate the result x2 after token interaction; then the output result x2 is added to the input x1 through a jump connection to obtain a new result x3; the new result x3 is normalized through the layer, and then input into the forward feedback network, and two linear transformations and ReLU nonlinear activation functions are performed to obtain the output feature result x4; the output result x4 is added to the input x3 through a jump connection to obtain the final result x5.

6. The image restoration system based on dense sampling and adaptive sparse sampling according to claim 1, characterized in that: The image reconstruction module includes: a preprocessing unit, an upsample unit (Upsample) and an output unit, wherein: the preprocessing unit performs a 3×3 convolution process according to the feature F2 output by the feature processing module to obtain a new feature; the upsample unit performs a 3×3 convolution process and a shuffle process according to the output result of the preprocessing unit to obtain the upsampled feature result; the output unit performs a 3×3 convolution process according to the upsampled feature result based on the super-resolution task or the denoising and compression artifact removal task to obtain the output high-quality image I HQ Or, based on the feature F2 output by the feature processing module as input, a 3×3 convolution is performed, and the processed result is connected to the input low-quality image I through a jump connection. LQ Add together to get the final high-quality image I HQ .

7. The image restoration method based on dense sampling and adaptive sparse sampling according to any one of claims 1 to 6, characterized in that: include: The first step is to select a single low-quality image from the commonly used image restoration test dataset and input it into the image feature extraction module. Specifically, a convolutional neural network is used to extract the image feature from the input image I LQ ∈R H×D×3 The first feature F0∈R is extracted from H×D×C , where: H is the length of the image, D is the width of the image, the default number of image channels is 3, and C is the number of new channels for extracting features; The second step is to transform the first feature F0∈R H×D×C Input feature perception module, after regularization processing, the corresponding query vector query∈R is extracted through convolutional neural network HD×C , key vector key∈R hw×C Sum value vector value∈R hw×3 After that, the attention matrix attn∈R is calculated HD×hw , update the value vector value through the attention matrix to obtain the new output features and sparse coefficients I; Step 3: The second feature F1 is processed by multiple residual module groups RG in sequence, and the processed features are processed through a mapping layer and 3×3 convolution to obtain output features, and the output features are added to the input features F1 through a jump connection to obtain the result F2; wherein: each RG module includes N Transformer submodules, the 2i-1th submodule is set as a dense sampling Transformer submodule based on a window, and the 2ith submodule is set as a sparse sampling Transformer submodule based on a variable coefficient I. After the features processed by N Transformer submodules pass through a layer of 3×3 convolutional neural network, they are added to the input features of the first submodule through a jump connection to obtain the output features of the RG module; The fourth step is to input the output feature F2 of the third step into the image reconstruction module to reconstruct a high-quality image. Specifically, for the super-resolution task, the second feature F2 is processed by a 3×3 convolution layer through the preprocessing unit to obtain the output feature F * =Conv 3×3 (F2); New output F * After the upsampling unit (Upsample), a layer of 3×3 convolution processing and shuffle processing are performed, and the result is further processed by the output unit and a layer of 3×3 convolution processing is performed to obtain a high-quality image: I HQ =Conv 3×3 (Upsample(F * )); For tasks such as denoising and decompression artifacts, the second feature F2 is passed through the output unit and subjected to a layer of 3×3 convolution. The result is connected to the input low-quality image I through a jump connection. LQ Add together to get a high-quality image: I HQ =Conv 3×3 (F2)+I LQ .

8. The image restoration method based on dense sampling and adaptive sparse sampling according to claim 7, characterized in that: The second step specifically includes: 2.1) Normalize the first feature F0 to obtain the intermediate feature Then, query∈R is obtained by a convolutional neural network with a convolution kernel size of 1×1, stride of 1, and padding of 0. HD×C , after being processed by a convolutional neural network with a convolution kernel size of 4×4, stride of 4, and padding of 0, the key vector key∈R is obtained hw×C Sum value vector value∈R hw×3 , where: h = H / / s, w = D / / s, / / means downsampling by s times; 2.2) Calculate the updated first feature Where: attention matrix attn = query × key T , where: attn∈R HD×hw , Softmax is the activation function, C is the number of channels of each head during self-attention interaction, and T is the transposition operation; 2.3) Using the attention matrix and the new activation function sigmoid, we get the activation value of the attention matrix. Using this information distribution and the threshold gate operation, we set the activation value less than 0.5 to be unimportant, that is, 0, and the activation value not less than 0.5 to 1. We count the number of 1s to get the activation rate and sparse coefficient, specifically: activation rate λ = sum(gate(sigmoid(attn)) / num(attn), sparse coefficient Among them: sum() is to sum the matrix, mum() is to count the number of elements in the matrix, and vs is the preset sparse initial value; 2.4) Calculate the second feature based on the intermediate feature and the updated first feature Among them: proj() is a linear layer neural network.

9. The image restoration method based on dense sampling and adaptive sparse sampling according to claim 7, characterized in that: In the third step, the window-based dense sampling Transformer submodule processing process includes: 3.1) The input features are normalized by layers to obtain the intermediate features F s , and then the H×D feature map is densely sampled, and the feature tokens are grouped and sampled according to the W×W window size. The number of tokens in each group is W×W, which are distributed in the dense area of ​​a certain window range of the original feature map. The updated intermediate feature Each group of token features F t ∈R WW×C ;in, That is F n By multiple F t Cascading along the first dimension; 3.2) Each group of token features F t After three linear layers, query, key and value are extracted, and the features after self-attention interaction are calculated. Among them, Softmax is the activation function, T is the transposition operation, and v is the number of channels of each head during self-attention interaction; 3.3) The feature F after self-attention interaction a After a linear layer, it is connected to the intermediate feature F through a skip connection. s After addition, the output is sent to the Feedforward Network (FFN) to obtain the final output features.

10. The image restoration method based on dense sampling and adaptive sparse sampling according to claim 7, characterized in that: In the third step, the sparse sampling Transformer submodule processing process based on the variable coefficient I includes: 3.a) The input features are normalized to obtain the intermediate features_F s Then, according to the sparse coefficient I obtained in the second step, the feature map of H×D is sampled in groups with an interval of I according to the sparse sampling method. The number of tokens in each group is Distributed in the sparse area of ​​the original feature map, the updated intermediate features are obtained The characteristics of each token group are in, m=I×I,that is, _F n By multiple _F t Cascading along the first dimension; 3.b) Each group of token features_F t After three linear layers, query, key and value are extracted, and the features after self-attention interaction are calculated. Among them, Softmax is the activation function, T is the transposition operation, and C is the number of channels of each head during self-attention interaction; 3.c) The feature after self-attention interaction_F a After being processed by a linear layer, the intermediate features are added through skip connection and normalization and then output to the feed-forward network (FFN) to obtain the final output features.

Images