Efficient feature learning method and system based on windowless Gaussian vision Transform

Abstract

The invention provides an efficient feature learning method and system based on windowless Gaussian vision Transform, and relates to the field of computer vision. According to the method, a windowless Gaussian self-attention module (WGSAB) is designed as a main construction block of a neural network WGViT, and tokens are sampled and interacted through two-dimensional Gaussian distribution in an image processing task, so that feature expression is not limited by a window, long-distance and short-distance dependence of a neural network captured image is realized, the calculation complexity of self-attention is reduced, and the image processing efficiency is improved. Efficient feature learning is carried out; a plurality of WGSAB network modules are stacked to form a neural network WGViT, Gaussian token sampling (GTS) in the WGSAB captures different long and short distance dependencies by setting different mean values, covariances and sampling rates, and the obtained neural network model can be used for trunks of visual tasks such as image classification, target detection, instance segmentation and semantic segmentation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an efficient feature learning method and system based on a windowless Gaussian vision Transformer. Background Art

[0002] Convolutional Neural Networks (CNNs) and Visual Transformers (ViTs) are two mainstream feature learning methods that have shown good prospects in multiple research and application fields. CNNs capture local features through sliding windows of convolution kernels, which limits their ability to capture global information. ViT can capture global information well, but has a high computational complexity, while local ViT reduces computational complexity through non-overlapping windows. This window limitation usually requires deeper network layers to effectively fuse features through cross-window interactions. To address these issues, many studies have focused on optimizing network structures to reduce computational complexity, such as designing efficient convolution operations and exploring ViT's sparse attention mechanism. The convolution operation uses large kernel convolutions to increase the receptive field and uses position offsets to break through the limitations of small convolution kernel windows. Similarly, ViT has adopted non-overlapping sub-windows and sparse attention mechanisms to improve efficiency without sacrificing the advantages of global receptive field. Summary of the invention

[0003] The present invention provides an efficient feature learning method based on windowless Gaussian visual Transformer, comprising the following steps: S1. Designing a windowless Gaussian self-attention module (WGSAB) as the main building block of the neural network, the feature expression in image processing tasks is not limited by the window, and the neural network can capture the long- and short-range dependencies of features and perform efficient feature representation learning; S2. Stacking multiple WGSAB network modules constitutes the neural network WGViT. WGViT includes four stages, each of which is composed of WGSAB. The GTS in WGSAB captures different long- and short-distance dependencies by setting different means, covariances, and sampling rates. Image downsampling operations are performed between stages to form a feature pyramid structure.

[0004] Preferably, the Gaussian self-attention module (WGSAB) in S1, WGSAB consists of convolutional position embedding (CPE), Gaussian token sampling (GTS), multi-head self-attention (MHSA) and feed-forward network (FFN). Given an input image , where C is the embedding dimension, H is the height of the input image, and W is the width of the input image. Inputting it into WGSAB can be represented by the following process: , , The first item indicates that the token sampled by GTS performs multi-head self-attention operation, and the second item indicates that the token not sampled by GTS is not processed. .

[0005] Preferably, for WGSAB in S1, WGSAB samples tokens through a two-dimensional Gaussian distribution and interacts so that the feature expression is not limited by the window. Specifically, GTS is used to input feature maps The tokens in are sampled using a two-dimensional Gaussian distribution Sampling is achieved, where , , , represents the mean position of the samples, randomly and uniformly selected from the feature map, , and Represents the variance of the x-axis and y-axis in a two-dimensional Gaussian distribution, with a sampling rate of ,in represents the sampling rate on the x-axis, Represents the sampling rate on the y-axis, and the token set obtained by sampling is , the number is .

[0006] Preferably, WGSAB in S1 only performs multi-head self-attention operation on the tokens sampled by GTS, so as to realize the long- and short-range dependencies of the image captured by the neural network, reduce the computational complexity of self-attention, and perform efficient feature representation learning;

[0007] Preferably, the neural network WGViT in S2 includes four stages, each of which is composed of WGSAB. Downsampling operations are performed on images between stages to form a feature pyramid structure. Each stage of WGViT adopts the WGSAB module, and each stage constructs different long- and short-distance dependencies based on the tokens obtained from GTS.

[0008] The present invention also provides an efficient feature learning system based on a windowless Gaussian visual Transformer, which is characterized by comprising: a windowless Gaussian self-attention module (WGSAB), which samples tokens through a two-dimensional Gaussian distribution and interacts with each other, so that the feature expression is not restricted by the window, and the neural network captures the long- and short-range dependencies of the image, reducing the computational complexity of self-attention; a stacking module, in which multiple WGSAB modules are stacked to form a four-stage neural network, and Gaussian token sampling (GTS) in each stage captures different long- and short-range dependencies by setting different means, covariances and sampling rate strategies, which is an efficient feature learning system.

[0009] Compared with the prior art, the present invention has the following technical effects: WGViT is a non-window-based visual Transformer that uses two-dimensional Gaussian distribution to sample tokens to achieve self-attention interaction, getting rid of window limitations, making feature expression more effective, and no longer relying on cross-window interactions to fuse features. Each stage of WGViT is composed of WGSAB, in which GTS sets different means, covariances, and sampling rates. The mean is randomly and uniformly selected from the feature map, and the covariance and sampling rate increase with the stage, taking into account the fusion of local and global features to achieve effective long- and short-distance dependencies. In WGSAB, self-attention calculations are only performed on tokens sampled by GTS, which significantly reduces the computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A neural network architecture diagram provided for an embodiment of the present invention; Figure 2 Configuration of a neural network family provided for an embodiment of the present invention; Figure 3 A diagram showing a comparison of the classification performance of the method of an embodiment of the present invention and several different methods on the ImageNet-1K dataset; Figure 4 Graph showing the performance comparison of target detection and instance segmentation of the method of an embodiment of the present invention and several different methods on the COCO val2017 dataset; Figure 5 This is a graph comparing the semantic segmentation performance of the method according to an embodiment of the present invention and several different methods on the ADE20K dataset. DETAILED DESCRIPTION

[0011] The present invention aims to provide an efficient feature learning method and system based on windowless Gaussian visual Transformer, designs a windowless Gaussian self-attention module (WGSAB) as the main building block of the neural network WGViT, samples tokens through two-dimensional Gaussian distribution and interacts in image processing tasks, so that feature expression is not restricted by windows, and the neural network captures the long- and short-range dependencies of images, reduces the computational complexity of self-attention, and performs efficient feature learning; stacking multiple WGSAB network modules constitutes the neural network WGViT, in which Gaussian token sampling (GTS) captures different long- and short-range dependencies by setting different means, covariances and sampling rates, and the obtained neural network model can be used as the backbone of visual tasks such as image classification, target detection, instance segmentation and semantic segmentation.

[0012] See also Figure 1As shown, an efficient feature learning neural network method and system based on a windowless Gaussian visual Transformer in an embodiment of the present application: S1. Designing Gaussian Self-Attention Block (WGSAB) as the main building block, the feature expression in image processing tasks is not limited by the window, which enables the neural network to capture the long- and short-range dependencies of features and perform efficient feature representation learning; S2. Stacking multiple WGSAB network modules constitutes the neural network WGViT. WGViT includes four stages, each of which is composed of WGSAB. In WGSAB, GTS captures different long- and short-distance dependencies by setting different means, covariances, and sampling rate strategies. Image downsampling operations are performed between stages to form a feature pyramid structure.

[0013] Furthermore, the Gaussian self-attention module (WGSAB) in S1, WGSAB consists of convolutional position embedding (CPE), Gaussian token sampling (GTS), multi-head self-attention (MHSA) and feed-forward network (FFN). Given an input image , where C is the embedding dimension, H is the height of the input image, and W is the width of the input image. Inputting it into WGSAB can be represented by the following process: , , The first item indicates that the token sampled by GTS performs multi-head self-attention operation, and the second item indicates that the token not sampled by GTS is not processed. First, the position information is encoded into This is a learnable position embedding that encodes the position through residual connections. Add to the original value to get the feature map , and then use GTS to get the feature map Sampling token, the sampled token is recorded as , represents the number of tokens sampled, C is the embedding dimension, these tokens are input to the MHSA module, MHSA is multi-head attention, and then the multi-head attention operation is connected through residual connection Add the original value to get the new representation of the sampled token, and output the remaining unsampled tokens directly to get the new feature representation. ,Will Input FFN to achieve cross-channel interaction. FFN consists of two linear layers, and a GELU function is added between the two linear layers. Finally, the residual connection is used to connect the FFN Add to the original value , This is the output of WGSAB.

[0014] WGSAB in S1 samples tokens through two-dimensional Gaussian distribution and interacts with them, so that feature expression is not limited by the window. The Gaussian token sampling module (GTS) in WGSAB is used to sample the input feature map The tokens in are sampled using a two-dimensional Gaussian distribution Sampling is achieved, where , , , represents the mean position of the samples, randomly and uniformly selected from the feature map, , and Represents the variance of the x-axis and y-axis in a two-dimensional Gaussian distribution, with a sampling rate of ,in represents the sampling rate on the x-axis, Represents the sampling rate on the y-axis, and the token set obtained by sampling is , the number is .

[0015] Furthermore, the neural network WGViT in S2,WGViT consists of four stages, each of which is composed of WGSAB. Downsampling operations are performed on the image between stages to form a feature pyramid structure. We instantiate WGViT with five different model sizes by setting different numbers of blocks, such as Figure 2 As shown in Figure 2, to obtain hierarchical feature maps, we use downsampling layers between stages to adjust the different scales of the feature maps. Figure 2 The Blocks in WGViT correspond to the number of blocks in the four stages. The number of blocks in each stage is the number of blocks in WGSAB. The Sampling Rate represents the sampling rate set in the four stages of GTS. The sampling rate increases with the stage, and the covariance also increases with the sampling rate. Each stage of WGViT adopts a windowless Gaussian self-attention module, which fully captures different long- and short-distance dependencies in each stage.

[0016] This embodiment provides an efficient feature learning system based on windowless Gaussian visual Transformer, characterized in that the windowless Gaussian self-attention module (WGSAB) samples tokens through two-dimensional Gaussian distribution and interacts, so that the feature expression is not limited by the window, and the neural network captures the long- and short-distance dependencies of the image, reducing the computational complexity of self-attention; the stacking module, multiple WGSAB modules are stacked to form a four-stage neural network, and the Gaussian token sampling (GTS) in each stage captures different long- and short-distance dependencies by setting different means, covariances and sampling rate strategies, which is an efficient feature learning system based on windowless Gaussian visual Transformer. The obtained neural network model can be used as the backbone of visual tasks such as image classification, target detection, instance segmentation and semantic segmentation.

[0017] The above are only preferred embodiments of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. An efficient feature learning method based on windowless Gaussian visual transformer, characterized in that: The following steps are involved: S1. Designing a windowless Gaussian self-attention module (WGSAB) as the main building block of the neural network, the feature expression in image processing tasks is not limited by the window, and the neural network can capture the long- and short-range dependencies of features and perform efficient feature representation learning; S11. For the Gaussian self-attention module (WGSAB), WGSAB consists of convolutional position embedding (CPE), Gaussian token sampling (GTS), multi-head self-attention (MHSA) and feed-forward network (FFN). Given an input image , where C is the embedding dimension, H is the height of the input image, and W is the width of the input image. Inputting it into WGSAB can be represented by the following process: , , ,The first item indicates that the token sampled by GTS performs multi-head self-attention operation, and the second item indicates that the token not sampled by GTS is not processed. ; S12, WGSAB samples tokens through two-dimensional Gaussian distribution and interacts with them, so that feature expression is not limited by the window. Specifically, GTS is used to input feature maps The token in adopts a two-dimensional Gaussian distribution Sampling is performed, where , , , represents the mean position of the samples, , and Represents the variance of the x-axis and y-axis in a two-dimensional Gaussian distribution, with a sampling rate of ,in represents the sampling rate on the x-axis, Represents the sampling rate on the y-axis, and the token set obtained by sampling is , the number is ; In S13 and WGSAB, only the tokens sampled by GTS are subjected to multi-head self-attention operation, so that the neural network can capture the long- and short-range dependencies of the image, reduce the computational complexity of self-attention, and perform efficient feature representation learning; S2. Stacking multiple WGSAB network modules constitutes the neural network WGViT. WGViT includes four stages, each of which is composed of WGSAB. In WGSAB, GTS captures different long- and short-distance dependencies by setting different means, covariances, and sampling rate strategies. Image downsampling operations are performed between stages to form a feature pyramid structure.

2. According to claim 1, an efficient feature learning method based on windowless Gaussian visual transformer is characterized in that: The neural network WGViT in S2 consists of four stages. Each stage is composed of WGSAB. Image downsampling operations are performed between stages to form a feature pyramid structure. Each stage of WGViT is a WGSAB module. Each stage constructs different long- and short-distance dependencies based on the tokens sampled by GTS.

3. An efficient feature learning system based on windowless Gaussian visual transformer, characterized in that: include: The windowless Gaussian self-attention module (WGSAB) samples tokens through two-dimensional Gaussian distribution and interacts with them, so that the feature expression is not restricted by the window, and the neural network can capture the long- and short-range dependencies of the image, reducing the computational complexity of self-attention; the stacking module, multiple WGSAB modules are stacked to form a four-stage neural network, and the Gaussian token sampling (GTS) in each stage captures different long- and short-range dependencies by setting different means, covariances and sampling rate strategies, that is, an efficient feature learning system that implements any of the methods described in claims 1-2.

Images