Image restoration system and method based on dense sampling and adaptive sparse sampling

Abstract

The application discloses an image restoration system and method based on dense sampling and adaptive sparse sampling, which comprises an image feature extraction module, a feature perception module, a Transform feature processing module based on dense sampling and adaptive sparse sampling and an image reconstruction module. According to the image internal representation information, the application can simultaneously extract local feature information and global feature information of the image, effectively adjust the correlation between the features while ensuring the calculation efficiency, reconstruct a high-quality image and significantly improve the model performance.

Description

Technical Field

[0001] This invention relates to a technique in the field of image processing, specifically an image restoration system and method based on dense sampling and adaptive sparse sampling. Background Technology

[0002] Existing image restoration methods based on the Transformer architecture typically employ window-based dense sampling or fixed-interval sparse sampling to sample the query vector, key vector, and value vector when designing their self-attention mechanisms. While these two methods can reduce computational complexity, they do not take into account the intrinsic information representation of features, and sparse sampling cannot adaptively adjust the sampling range to extract image features, thus limiting their improvement on model performance. Summary of the Invention

[0003] This invention addresses the problem that existing technologies do not significantly improve model performance. It proposes an image restoration system and method based on dense sampling and adaptive sparse sampling, which can simultaneously extract local and global feature information of an image based on its internal representation information. This method can adjust the correlation between features more effectively while ensuring computational efficiency, reconstruct high-quality images, and significantly improve model performance.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to an image restoration system based on dense sampling and adaptive sparse sampling, comprising: an image feature extraction module, a feature perception module, a Transformer feature processing module based on dense sampling and adaptive sparse sampling, and an image reconstruction module, wherein: the image feature extraction module uses an input low-quality image I... LQ ∈R H×D×3 Image feature extraction is performed to obtain the first feature F0∈R. H×D×C The feature perception module, based on the first feature F0, performs internal feature information perception and calculation processing to obtain the updated second feature F1∈R. H×D×C The Transformer feature processing module performs intra-feature token attention interaction processing based on the second feature F1 and the learned variable sparsity coefficients I, resulting in the updated feature F2∈R. H×D×C The image reconstruction module reconstructs a high-quality image I based on the processed feature F2. HQ .

[0006] The Transformer feature processing module comprises K residual module groups (RG) and a mapping layer (Mapping), wherein: the residual module group receives the output result F1 of the feature perception module, and is cyclically processed until the residual module group quantity meeting the processing reaches K, to obtain a processed image feature; the mapping layer performs one layer of 3*3 convolution processing on the image feature to obtain a mapped feature F map ; the mapped feature F map and the initially input feature F1 are added through a jump connection to obtain an updated result feature F2.

[0007] The residual module group comprises N dense sampling Transformer submodules (DTB), adaptive sparse sampling Transformer submodules (sTB) and a mapping layer, wherein: the 2i-1th submodule is set as a dense sampling Transformer submodule, the 2ith submodule is set as an adaptive sparse sampling Transformer submodule, and the last processing unit is set as a mapping layer, wherein: the ith submodule receives the output result of the i-1th submodule, performs internal feature processing to obtain a new image feature, and the processing is cyclically performed until the number of processed Transformer submodules reaches N; the mapping layer performs one layer of 3*3 convolution processing on the processed image feature to obtain a mapped feature F m , and then the mapped feature F m and the input feature F in of the first submodule are added through a jump connection to obtain an output feature F out .

[0008] The dense sampling Transformer submodule comprises a layer normalization (LayerNorm) unit, a dense sampling unit, a multi-head self-attention mechanism (MSA) unit and a forward feedback (FFN) unit, wherein: the layer normalization unit performs unbiased processing on each layer channel dimension feature value of an image according to input x1 to obtain a result; the dense sampling unit performs dense sampling on the feature map distribution by dividing a window according to the layer normalization result to obtain a resampling sequence token; the multi-head self-attention mechanism unit extracts multiple groups of query, key and value vectors according to the resampling sequence token to calculate a token interaction result x2; then the output result x2 is added to the input x1 through a jump connection to obtain a new result x3; the new result x3 is subjected to layer normalization, and then is subjected to two times of linear transformation and ReLU nonlinear activation function processing through a forward feedback network to obtain an output feature result x4; the output result x4 is added to the input x3 through a jump connection to obtain a final result x5.

[0009] The adaptive sparse sampling Transformer sub-module comprises a layer normalization (LayerNorm) unit, an adaptive sparse sampling unit, a multi-head self-attention mechanism (MSA) unit and a feed-forward feedback (FFN) unit, wherein: the layer normalization unit normalizes each layer channel dimension feature value of an image according to input x1 to obtain an unbiased result; the adaptive sparse sampling unit uses sparse coefficients I output by a feature perception module to extract tokens every I steps on a feature map to perform sparse sampling on the feature map distribution and obtain a resampling sequence token according to the layer normalization result; the multi-head self-attention mechanism unit calculates the result x2 after token interaction according to the resampling sequence token; then the output result x2 is added to the input x1 through a skip connection to obtain a new result x3; the new result x3 is subjected to layer normalization, then input into a feed-forward feedback network to perform two linear transformations and ReLU nonlinear activation functions to obtain a feature result x4 after output; the output result x4 is added to the input x3 through a skip connection to obtain a final result x5.

[0010] The image reconstruction module comprises a preprocessing unit, an up-sampling unit (Upsample) and an output unit, wherein: the preprocessing unit performs 3*3 convolution processing on the feature F2 output by the feature processing module to obtain a new feature; the up-sampling unit performs 3*3 convolution processing and shuffle processing once on the output result of the preprocessing unit to obtain a feature result after up-sampling; the output unit performs 3*3 convolution processing once on the feature result after up-sampling to obtain a high-quality image I HQ or performs 3*3 convolution processing once on the feature F2 output by the feature processing module as input, and adds the processing result to the input low-quality image I LQ through a skip connection to obtain the final high-quality image I HQ .

[0011] Technical effects

[0012] The present application utilizes the internal correlation between features, acquires effective information inside the features, acquires sparse coefficients through feature perception, utilizes window-based dense sampling and sparse sampling based on sparse coefficients in the feature processing stage, processes the features through a self-attention interaction unit, adaptively extracts local information and global information of an image, and further improves the quality of the generated image in the final image reconstruction stage. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 It is a schematic diagram of the system of the present application;

[0014] Figure 2This is a schematic diagram of the feature perception module;

[0015] Figure 3 This is a schematic diagram of the feature processing module;

[0016] Figure 4 This is a schematic diagram of the image reconstruction module. Detailed Implementation

[0017] like Figure 1 As shown, this embodiment illustrates an image restoration system based on dense sampling and adaptive sparse sampling. The system includes: an image feature extraction module, a feature perception module, a Transformer feature processing module based on dense sampling and adaptive sparse sampling, and an image reconstruction module. The image restoration method includes:

[0018] The first step is to select a single low-quality image from a commonly used image restoration test dataset as the input image feature extraction module. Specifically, this involves using a convolutional neural network to extract features from the input image I. LQ ∈R H×D×3 The first feature F0∈R is extracted. H×D×C Where: H is the length of the image, D is the width of the image, the default number of image channels is 3, and C is the number of new channels for feature extraction;

[0019] The convolutional neural network is a single layer with a kernel size of 3×3.

[0020] The second step is to assign the first feature F0∈R H×D×C The input feature perception module, after regularization, extracts the corresponding query vector query∈R through a convolutional neural network. HD×C Key vector key∈R hw×C Sum of value vectors value∈R hw×3 Then, the attention matrix attn∈R is calculated. HD×hw The attention matrix is ​​used to update the value vector to obtain new output features and sparse coefficients I, specifically including:

[0021] 2.1) The first feature F0 is normalized to obtain the intermediate features. Then, the query∈R is obtained by processing it through a convolutional neural network with a kernel size of 1×1, a stride of 1, and padding of 0. HD×C The key vector key∈R is obtained by processing the convolutional neural network with a kernel size of 4×4, a stride of 4, and padding of 0. hw×C Sum of value vectors value∈R hw×3 Where: h = H / / s, w = D / / s, / / indicates downsampling by s times;

[0022] The down-sampling multiple s is preferably a multiple of 4.

[0023] 2.2) Calculate the updated first feature Wherein: attention matrix attn=queryxkey T , wherein: attnR HD×hw , Softmax is an activation function, C is the number of channels per head when self-attention interaction, T is the transpose operation;

[0024] The number of heads when calculating attention interaction is preferably 6.

[0025] 2.3) Using the attention matrix, combined with the new activation function sigmoid, get the activation value of the attention matrix, use these information distribution combined with the threshold gate operation, set the activation value less than 0.5 as unimportant, that is 0, otherwise, set the activation value not less than 0.5 as 1, count the number of 1 to get the activation rate and sparse coefficient, specifically: activation rate λ=sum(gate(sigmoid(attn)) / num(attn), sparse coefficient Wherein: sum() is the sum of the matrix, num() is the number of elements in the matrix, vs is the preset sparse initial value.

[0026] The attention matrix attnR HD×hw The information contained therein represents the degree of correlation between image pixels, and after activation by the sigmoid function, the correlation between image pixel information is mapped to the 0-1 interval, so further filter out the activation value less than 0.5 through the threshold gate operation, so the image pixel information retained is the pixel point with strong representation ability, by counting the percentage of these pixel points in the total number of image pixels, to judge the overall representation information of the image, the stronger the correlation between pixels, the more activated pixel points, so the greater the activation rate and the smaller the sparse coefficient.

[0027] The sparse initial value can be adjusted according to different data sets, and the default value is 4.

[0028] 2.4) Calculate the second feature according to the intermediate feature and the updated first feature Wherein: proj() is a linear layer neural network.

[0029] Third step: the second feature F1 is sequentially processed by multiple residual module groups RG, and the processed feature is processed by a mapping layer to obtain an output feature through 3x3 convolution processing. The output feature is added to the input feature F1 through a jump connection to obtain a result F2. Wherein: each RG module includes N Transformer sub-modules, the 2i-1th sub-module is set as a window-based dense sampling Transformer sub-module, and the 2i th sub-module is set as a sparse sampling Transformer sub-module based on a variable coefficient I. After the features processed by the N Transformer sub-modules are processed by a 3x3 convolutional neural network, the output feature of the RG module is obtained by adding the input feature of the first sub-module through a jump connection.

[0030] The window-based dense sampling Transformer sub-module processing process includes:

[0031] 3.1) The input feature is processed by layer normalization to obtain an intermediate feature F s The HxD feature map is sampled in a dense sampling manner according to the window size of WxW, and each group of token features F is obtained. t ∈R WW×C . Wherein, F n is obtained. t is obtained.

[0032] 3.2) Each group of token features F t is extracted through three linear layers to obtain query, key and value, and the feature after self-attention interaction is calculated. Wherein, Softmax is an activation function, T is a transpose operation, and C is the number of channels of each head during self-attention interaction.

[0033] 3.3) The feature F a after self-attention interaction is processed by a linear layer, and then added to the intermediate feature F s through a jump connection and output to a forward feedback network (FFN) to obtain the final output feature.

[0034] The FFN network is composed of two linear networks and a Relu activation unit, and the output feature of the FFN network is added to the input feature of the FFN through a jump connection to obtain the final output feature result.

[0035] The processing process of the sparse sampling Transformer sub-module based on the variable coefficient I includes:

[0036] 3. a) the input features are normalized to obtain intermediate features_F s According to the sparse coefficient I obtained in the second step, the feature map of HxD is sampled in a sparse sampling manner with an interval of I to group and sample the token features, and the number of tokens in each group is The sparse region in the original feature map is distributed to obtain updated intermediate features

[0037] Each group of token features is Where, m=I*I, that is, the_F n A plurality of_F t are concatenated along the first dimension.

[0038] 3. b) each group of token features_F t is extracted through three linear layers to calculate the features after self-attention interaction Where Softmax is an activation function, T is a transpose operation, and C is the number of channels of each head during self-attention interaction.

[0039] 3. c) the features_F a after self-attention interaction are processed through a linear layer, added to the intermediate features through a jump connection and normalization, and then output to a forward feedback network (FFN) to obtain the final output features.

[0040] The FFN network consists of two linear networks and a Relu activation unit, wherein the output features of the FFN network are added to the input features of the FFN through a jump connection to obtain the final output feature result.

[0041] In the fourth step, the output features F2 of the third step are input into the image reconstruction module to reconstruct a high-quality image, specifically: for a super-resolution task, the second feature F2 is processed through a preprocessing unit to obtain output features F * =Conv 3×3 (F2); the new output F * is processed through an upsampling unit (Upsample) to obtain a result, which is further processed through an output unit to obtain a high-quality image I HQ =Conv 3×3 (Upsample(F *); for the task of denoising, de-compression artifact, etc., the second feature F2 is processed by the output unit to obtain a result through a 3x3 convolution, and the result is connected to the input low-quality image I LQ The high-quality image I is obtained by adding HQ =Conv 3×3 (F2)+I LQ .

[0042] The upsampling is implemented by, but not limited to, the sub-pixel convolutional neural network described in Shi, Wenzhe, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[0043] Through specific actual experiments, the GPU model NVIDIA Ge Force RTX 3090 24GB is used in the hardware aspect, the system version of the server is Ubuntu 18.04.6 LTS in the software environment aspect, the CUDA version is 11.6, all the related codes involved in the experiment are written by Python language, the version is 3.8.11, and the machine learning library used when the neural network is built is provided by Pytorch. 1) The super-resolution multiple is set to 2, the training data is DIV2K and Flickr2K, a total of 3550 high-resolution images and 3550 low-resolution images down-sampled by bicubic linear interpolation by two times, and the test data is the public five data sets Set5, Set14, B100, Urban100 and Manga109. 2) The preset parameters are: the channel number C of feature extraction is 180, the number of residual module groups RG is 4, the number of Transformer modules in each RG is 3 DTB modules and 3 STB modules, the number of heads is 6 when attention interaction, the window size of window-based dense sampling is 8. 3) In the experiment process, the training data is trained for 500000 steps, the L1 loss is used for training, the Adam optimizer is used, the initial learning rate is 2e-4, then the learning rate is reduced by half at steps [250000, 400000, 450000, 475000], the trained model is obtained, the PSNR and SSIM indicators are tested on the five test sets, and the obtained indicator results are shown in Table 1.

[0044] Table 1

[0045] Indicator Set5 Set14 B100 Urban100 Manga109 PSNR 38.3856 34.3636 32.4870 33.7451 39.9291 SSIM 0.9620 0.9251 0.9036 0.9417 0.9799

[0046] The control experiment uses the same network structure as the network built using the technology, the only difference is that the feature perception module is abandoned to extract the information of the internal correlation of the features, and the subsequent variable sparse coefficient is not used for sparse sampling. The sampling range is fixed during sparse sampling, and the sampling interval I is set to 4. The experimental process is based on the same training data and trained for 500,000 steps under the same experimental parameter settings. The trained model is tested on the five test sets for the indicators PSNR and SSIM, and the obtained indicator results are shown in Table 2.

[0047] Table 2

[0048] Indicator set5 Set14 B100 U100 M109 PSNR 38.3132 34.2498 32.4418 33.4685 39.7416 SSIM 0.9618 0.9244 0.9031 0.9398 0.9793

[0049] Comparing Table 1 and Table 2, it is found that after using the technology related to the present application, the two commonly used performance indicators of the image restoration model on the five public data sets of the super-resolution task are improved. The reason is that the feature perception module learns the internal correlation information of the features, obtains the sparse coefficient to guide the adaptive sparse sampling of the feature processing module, and then learns more effective feature information to guide the feature reconstruction module to obtain higher quality images. While improving the performance, the adaptive sparse coefficient is used for sparse sampling, which can save more computing resources and improve the computing efficiency compared with the fixed interval sparse sampling.

[0050] The above specific implementation can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application. The protection scope of the present application is subject to the claims and is not limited by the above specific implementation. Each implementation within the scope is subject to the constraints of the present application.

Claims

1. An image restoration system based on dense sampling and adaptive sparse sampling, characterized in that, Comprising: The image feature extraction module, the feature perception module, the Transformer feature processing module based on dense sampling and adaptive sparse sampling, and the image reconstruction module, wherein: the image feature extraction module performs image feature extraction processing according to an input low-quality image , to obtain first features , the feature perception module performs feature internal information perception and calculation processing according to the first features , to obtain updated second features and learned variable sparse coefficients I; the Transformer feature processing module performs feature internal token attention interaction processing according to the second features and the variable sparse coefficients I, to obtain updated result features , and the image reconstruction module performs reconstruction according to the processed result features , to obtain a high-quality image ; The Transformer feature processing module comprises K residual module groups (RGs) and a mapping layer (Mapping), wherein the residual module groups receive output results of the feature perception module , are cyclically processed until the number of residual module groups that meet the processing requirement reaches K, and processed image features are obtained ; the mapping layer performs one layer of 3*3 convolution processing on the image features to obtain mapped features ; and the mapped features and the initially input features are added through a skip connection to obtain updated result features . The residual module group comprises: a dense sampling Transformer sub-module (DTB), an adaptive sparse sampling Transformer sub-module (STB), and a mapping layer, with a total of N modules. The (2i-1)th sub-module is set as a dense sampling Transformer sub-module, the 2i-th sub-module is set as an adaptive sparse sampling Transformer sub-module, and the last processing unit is set as a mapping layer. The ith sub-module receives the output of the (i-1)th sub-module, performs internal feature processing to obtain new image features, and this process is repeated until the number of Transformer sub-modules processing features reaches N. The mapping layer performs a 3×3 convolution operation based on the processed image features to obtain the mapped features. The features after mapping and the input features of the first submodule The output features are obtained by adding the features through skip connections. ; The image reconstruction module includes a preprocessing unit, an upsampling unit, and an output unit, wherein the preprocessing unit calculates the features output by the feature processing module. The preprocessing unit performs a 3×3 convolution to obtain new features; the upsampling unit performs a 3×3 convolution and a shuffle operation based on the output of the preprocessing unit to obtain the upsampled feature results; the output unit performs a 3×3 convolution based on the upsampled feature results, either for a super-resolution task or for denoising and decompression artifact removal tasks, to obtain a high-quality output image. Or based on the features output by the feature processing module As input, a 3×3 convolution is performed, and the result is connected to the low-quality input image via skip connections. Add them together to get the final high-quality image. .

2. The image restoration system based on dense sampling and adaptive sparse sampling according to claim 1, characterized in that, The dense sampling Transformer submodule includes: a Layer Normalization (LayerNorm) unit, a dense sampling unit, a Multi-Head Self-Attention (MSA) unit, and a Feedforward (FFN) unit, wherein: the Layer Normalization unit adjusts the input... The image's feature values ​​for each channel dimension are normalized to obtain an unbiased result. The dense sampling unit, based on the layer normalization result, performs dense sampling on the feature map distribution by dividing the window, obtaining a resampled sequence token. The multi-head self-attention mechanism unit extracts multiple sets of query, key, and value vectors from the resampled sequence token to calculate the result after token interaction. The result will then be output. Through jump connections and input Adding them together yields a new result. New results After layer normalization, followed by a feedforward network, two linear transformations and ReLU nonlinear activation function processing are performed to obtain the output feature result. Output results Through jump connections and input Add them together to get the final result. .

3. The image restoration system based on dense sampling and adaptive sparse sampling according to claim 1, characterized in that, The adaptive sparse sampling Transformer submodule includes: a Layer Normalization (LayerNorm) unit, an adaptive sparse sampling unit, a multi-head self-attention (MSA) unit, and a feedforward (FFN) unit, wherein: the Layer Normalization unit adjusts the input according to the layer normalization unit. The image's feature values ​​for each channel dimension are normalized to obtain an unbiased result. The adaptive sparse sampling unit, based on the layer normalization result and using the sparse coefficients I output by the feature perception module, extracts tokens every I steps on the feature map, performing sparse sampling on the feature map distribution to obtain a resampled token sequence. The multi-head self-attention mechanism unit, based on the resampled token sequence, extracts multiple sets of query, key, and value vectors to calculate the result after token interaction. The result will then be output. Through jump connections and input Adding them together yields a new result. New results After layer normalization, the input is then fed into a feedforward network, where it undergoes two linear transformations and a ReLU nonlinear activation function to obtain the output feature result. Output results Through jump connections and input Add them together to get the final result. .

4. The image restoration method based on dense sampling and adaptive sparse sampling of the system according to any one of claims 1-3, characterized in that, Comprising: The first step is to select a single low-quality image from a commonly used image restoration test dataset as an input image feature extraction module, specifically: using a convolutional neural network to extract first features from the input image wherein H is the length of the image, D is the width of the image, the default number of image channels is 3, and C is the new number of extracted feature channels.​ Second step, the first feature The input feature perception module is processed through the convolutional neural network to extract the corresponding query vector , key vector And value vector After that, the attention matrix is calculated The value vector is updated through the attention matrix To get new output features and sparse coefficients I; Step 3: the second feature After being processed by the multiple residual module groups RG in turn, the processed feature is processed by a mapping layer, 3x3 convolution processing is performed to obtain an output feature, and the output feature is added to the input feature through a jump connection to obtain a result ; wherein: each RG module includes N Transformer sub-modules, the 2i-1th sub-module is set as a window-based dense sampling Transformer sub-module, the 2i th sub-module is set as a variable coefficient I-based sparse sampling Transformer sub-module, and after the features processed by the N Transformer sub-modules are processed by a layer of 3x3 convolutional neural network, the output feature of the RG module is obtained by adding the jump connection and the first sub-module input feature. In the fourth step, the output feature of the third step is input into the fourth step The input image reconstruction module reconstructs a high-quality image, specifically: for a super-resolution task, the second feature is input into a preprocessing unit for one layer of 3x3 convolution processing to obtain an output feature ; the new output is input into an up-sampling unit (Upsample) for one layer of 3x3 convolution processing and shuffle processing, and the result is further input into an output unit for one layer of 3x3 convolution processing to obtain a high-quality image: ; for a denoising, compression artifact removal, or the like task, the second feature is input into an output unit for one layer of 3x3 convolution processing, and the result is added to the input low-quality image through a skip connection to obtain a high-quality image: .

5. The image restoration method based on dense sampling and adaptive sparse sampling according to claim 4, characterized in that, The second step specifically comprises: 2.1) to the first feature The normalized processing obtains the intermediate feature , and then the convolutional neural network processing with the convolution kernel size of 1x1, stride of 1 and padding of 0 obtains , and the convolutional neural network processing with the convolution kernel size of 4x4, stride of 4 and padding of 0 obtains the key vector and the value vector , wherein: , / / indicates down-sampling s times; 2.2) Calculate the updated first feature wherein: attention matrix wherein: Softmax is the activation function, C is the number of channels per head when self-attention interaction, T is the transpose operation; 2.3) Using the attention matrix, combined with the new activation function sigmoid, get the activation value of the attention matrix, use the information distribution combined with the threshold gate operation, set the activation value less than 0.5 as unimportant, that is, 0, and vice versa, set the activation value not less than 0.5 as 1, count the number of 1s to get the activation rate and the sparse coefficient, specifically: activation rate , sparse coefficient , where: sum() is the sum of the matrix, num() is the number of elements in the matrix, and vs is the preset sparse initial value; 2.4) computing the second feature according to the intermediate feature and the updated first feature wherein: proj() is a linear layer neural network.

6. The image restoration method based on dense sampling and adaptive sparse sampling according to claim 5, characterized in that, In the third step, the dense sampling Transformer submodule processing process based on the window comprises: 3.1) Obtain intermediate features by performing layer normalization on the input features. Then, the H×D feature map is sampled densely, with feature tokens grouped into W×W windows. Each group contains W×W tokens, distributed within a dense region of a certain window in the original feature map, resulting in updated intermediate features. Each set of tokens has the following characteristics. ;in, , ,Right now Composed of multiple Obtained by cascading along the first dimension; 3.2) Calculate the token features of each group The query, key, and value are extracted through three linear layers, and the features after self-attention interaction are calculated where Softmax is the activation function, T is the transpose operation, and C is the number of channels for each head during self-attention interaction; 3.3) the features after self-attention interaction After processing by a linear layer, the output is added to the intermediate features through a skip connection and output to a feed-forward network (FFN) to obtain the final output features.

7. The method of claim 5, wherein the method is characterized by, In the third step, the sparse sampling Transformer submodule processing process based on the variable coefficient I comprises:

3. a) input features are normalized to obtain intermediate features According to the sparse coefficient I obtained in the second step, the feature map of Hx D is sampled in a sparse sampling manner with an interval of I, and the number of tokens in each group is , which is distributed in the sparse area of the original feature map to obtain the updated intermediate features , and each group of token features is ; wherein , , that is obtained by cascading a plurality of along the first dimension 3.b) the token features of each group are concatenated The query, key and value are extracted through three linear layers, and the features after self-attention interaction are calculated wherein Softmax is an activation function, T is a transpose operation, and C is the number of channels of each head during self-attention interaction. 3.c) the features after self-attention interaction After processing through a linear layer, the intermediate features are added to the output of the forward feedback network (FFN) through a jump connection and normalization processing to obtain the final output features.

Images