A novel multi-granularity feature fusion method based on attention mechanism

Abstract

The application provides a novel multi-granularity feature fusion method based on an attention mechanism, taking fine-granularity features obtained by a convolution network, block-granularity features constructed by a windowed MLP and global-granularity features extracted by an MLP as processing objects, constructing feature fusion modules between two different granularities according to the characteristics of the block-granularity and global-granularity features and the characteristics of the coarse-granularity and fine-granularity features, introducing the attention mechanism into the fusion process of different-granularity features by using the characteristics of different-granularity features, completing the process of transforming the global-granularity features into attention weights, realizing the fusion of the window block-granularity features and the global-granularity features, extracting the attention weights from the fusion features of the window block-granularity features and the global-granularity features, and realizing the filtering and edge strengthening of the fine-granularity features, and realizing the function of efficiently fusing different-granularity features of the convolution network, the Transformer or the MLP network.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision semantic segmentation, and particularly relates to a novel multi-granularity feature fusion method based on an attention mechanism. BACKGROUND

[0002] In the past decade, convolutional neural networks have made great progress in various fields of image processing (such as image recognition, object detection, semantic segmentation), and classic network models such as ResNet, YOLO, and FCN have emerged. Although convolutional neural networks have strong feature learning capabilities, due to the size limitation of the convolution kernel, convolutional neural networks can only construct fine-grained local features between pixels based on fixed-size convolution kernels, and cannot establish global relationships for each pixel in the feature map. Some studies have improved the problem of the locality of convolutional networks by introducing methods such as dilated convolution, feature pyramid, and self-attention mechanism. However, due to the limitations of convolutional networks, these studies still cannot effectively extract global context features.

[0003] The Transformer architecture based on full self-attention mechanism can establish global relationships between each position in the feature map, and has been widely studied in the field of natural language processing. In the past few years, visual Transformer (ViT) has achieved global (coarse-grained) relationships between each position in the feature map by blocking the input image and using full self-attention operations. However, due to the lack of prior bias of convolution operations in ViT, and the complexity of establishing long-range dependencies, ViT cannot be directly used in computer vision tasks that require real-time performance. Subsequent works have addressed these two issues, such as Swin Transformer and DeiT, but the running efficiency still cannot meet the real-time requirements. In recent years, researchers have found that using only a hybrid multi-layer perceptron (MLP) can also establish global dependencies between features. Compared with ViT, the processing speed of the hybrid MLP is faster.

[0004] In summary, convolutional networks have strong modeling capabilities for local features, and Transformers and MLPs can model global long-range dependencies. Current research has found that networks constructed by mixing convolution operations and Transformers or MLPs can achieve a better balance between efficiency and performance compared to convolutional networks and Transformer or MLP networks. However, the main problem that arises when mixing the two operations is how to efficiently fuse the fine-grained features extracted by convolutional networks and the coarse-grained features extracted by MLPs or Transformers. Typical methods include directly adding, multiplying, or concatenating the two different granularity features, but these operations ignore the differences and correlations between different features. SUMMARY

[0005] The technical problem solved by the present application is to provide a novel multi-granularity feature fusion method based on an attention mechanism for efficiently fusing different granularity features of convolutional networks, Transformers or MLP networks.

[0006] The technical solution adopted by the present application to solve the above technical problem is a novel multi-granularity feature fusion method based on an attention mechanism, comprising the following steps:

[0007] S0: building a novel multi-granularity feature fusion system based on an attention mechanism, including an encoding network, a decoding network, a first fusion module and a second fusion module; the first fusion module is used to fuse global granularity feature maps and block granularity feature maps, and output coarse-granularity feature maps; the second fusion module is used to fuse coarse-granularity feature maps and fine-granularity feature maps, and output final fusion feature maps;

[0008] S1: inputting an image into the encoding network and using convolutional operations to extract fine-granularity feature maps f c of the image, with a feature scale of 2Hx2WxC;

[0009] S2: inputting the features obtained through convolutional operations into a windowed multilayer perceptron W-MLP to extract block granularity feature maps f m of the image, with a feature scale of 2Hx2WxC;

[0010] S3: inputting the features obtained through convolutional operations into a multilayer perceptron MLP to extract global granularity feature maps f g ; the feature scale of the global granularity feature maps f g is HxWxC';

[0011] S4: inputting the global granularity feature maps f g and the block granularity feature maps f m into the first fusion module; let f w represent a global attention weight factor feature map, with a feature scale of HxWx1xC; let f b represent fusion feature maps after global granularity feature enhancement, with a feature scale of 2Hx2WxC; let f u represent coarse-granularity feature maps, then the fusion process of the first fusion module is represented by the following formula:

[0012] f w =Reshape(Sigmoid(Conv1x1(f g ))) (1),

[0013] f m =Reshape(Pus(f m)) (2),

[0014] f b = Reshape (Ps (f m x f w + f w )) (3),

[0015] f u = Cat (LN (Up (f g )), LN (f b )) (4),

[0016] In the above formula, Conv1x1 represents a 1x1 convolution, Sigmoid represents a Sigmoid operation, Reshape represents a feature map dimension transformation, Pus represents a Pixel Unshuffle pixel reorganization operation, P s sh represents a Pixel shuffle operation, LN represents a layer normalization operation, Up represents a bilinear upsampling operation, and Cat represents a feature concatenation operation.

[0017] S5: input the coarse-grained feature map f u output by the first fusion module and the fine-grained feature map f c into a second fusion module; let f aw represent an attention weight feature map, and the feature scale be 2Hx2WxC; let f o represent an edge enhancement feature map; let f a represent a final fusion feature map; and the fusion process of the second fusion module is represented by the following formula:

[0018] f aw = Sigmoid (Conv1x1 (f u )) (5),

[0019] f o = Sobel (f aw x f c ) + f c (6),

[0020] f a = Cat (LN (f u ), LN (f o )) (7),

[0021] In the above formula, Sobel represents a Sobel edge extraction operation.

[0022] S6: input the feature maps output by each step of the first fusion module and the second fusion module into a decoding network, respectively, to output corresponding semantic segmentation results.

[0023] According to the above scheme, in step S4, the first fusion module fuses the global granularity feature map f g Particle size characteristic map f m The specific steps are as follows:

[0024] S41: The first fusion module processes the global granular feature map f g Perform a 1x1 convolution followed by a Sigmoid operation sequentially to obtain the global granularity feature map f. g The attention weight coefficients at each position are calculated; the output features are then transformed to obtain a global attention weight factor feature map f containing semantic information. w ;

[0025] S42: For the block-grain size characteristic map f m A pixel unshuffle operation is performed to reduce the feature map spatial resolution to one-quarter of its original size and increase the channel dimension to four times its original size. Through dimensionality transformation, a block-scale feature map f is output. m The scale is H×W×4×C;

[0026] S43: Transfer the global attention weight factor feature map f w Compared with the block granularity feature map f after step S42 m Multiply them, and then combine the result with the global attention weight factor feature map f. w The features are added together to enhance the global semantic features; finally, after pixel shuffle and channel transformation operations, a fused feature map f with global granular feature enhancement is obtained. b ;

[0027] S44: Transfer the global granularity feature map f g Perform bilinear interpolation twice, then normalize the interpolated feature map by layer; then combine the fused feature map f after global granular feature enhancement. b Perform layer normalization; then, merge the two types of features output from the layer normalization operation using a concatenation operation to output a coarse-grained feature map f. u .

[0028] Furthermore, in step S5, the second fusion module fuses the coarse-grained feature map f. u With fine-grained feature map f c The specific steps are as follows:

[0029] S51: The second fusion module processes the coarse-grained feature map f u Performing 1x1 convolution and sigmoid operations sequentially yields an attention weight feature map f with semantic information and the same channel dimension as the fine-grained model. aw ;

[0030] S52: Convert the attention weight feature map f output from step S51 into a single image. aw With fine-grained feature map f c Multiply; then input the result of the multiplication into the Sobel edge extraction operator to extract the edge attention feature map; finally, connect the fine-grained feature map f via jumpers. c Adding it to the edge attention feature map yields an edge enhancement feature map f, which enhances the fine-grained features at the semantic level. o ;

[0031] S53: For coarse-grained feature maps f u and edge enhancement feature map f o Perform layer normalization operations separately; then merge the two types of features output from the layer normalization operations using a concatenation operation to output the final fused feature map f. a .

[0032] According to the above scheme, in step S4, the "windowed multilayer perceptron W-MLP" is replaced with the "windowed SwinTransformer network".

[0033] According to the above scheme, in step S5, "Multilayer Perceptron (MLP)" is replaced with "Transformer Network".

[0034] A computer storage medium storing a computer program executable by a computer processor, the computer program executing a novel multi-granularity feature fusion method based on an attention mechanism.

[0035] The beneficial effects of this invention are as follows:

[0036] 1. This invention discloses a novel multi-granularity feature fusion method based on an attention mechanism. It takes fine-grained features obtained from convolutional networks, block-grained features constructed by windowed MLP (or Swing Transformer), and global-grained features extracted by MLP (or Transformer) as processing objects. Targeting the characteristics of block-grained and global-grained features, as well as coarse-grained and fine-grained features, a feature fusion module between two different granularities is constructed. Utilizing the characteristics of different granularity features, an attention mechanism is introduced into the fusion process, completing the transformation of global-grained features into attention weights, the fusion of windowed block-grained features and global-grained features, the extraction of attention weights from the fused features, and the filtering and edge enhancement of fine-grained features. This achieves the function of efficiently fusing different granularity features from convolutional networks, Transformers, or MLP networks.

[0037] 2.The first fusion module of the present application utilizes the relationship between pixel points in each spatial position established by global features to enhance and suppress the semantic objects and background areas of block granularity features respectively. The first fusion module can fuse global granularity features at the block granularity level, ultimately making the block granularity feature map have more global semantic information. At the same time, by means of concatenation operation, global features are introduced into the input of the next layer network, which can better utilize global features to guide the fusion operation between different granularities. This module can be used for effective fusion between the block granularity features extracted by windowed Transformer or windowed MLP and the global granularity features extracted by Transformer or MLP network, improving the overall performance of pixel-intensive tasks such as semantic segmentation and object detection.

[0038] 3.The second fusion module of the present application utilizes the semantic information of coarse-grained features and the detail information of fine-grained features, and realizes the strengthening of semantic object areas while adding more detail information by introducing an attention mechanism. Then, the edge extraction operator is used to realize the enhancement of the edges of semantic objects. The two fusion modules proposed can be directly used for the fusion between different granularity features extracted by convolutional networks and MLP or Transformer networks, thereby improving the overall performance of pixel-intensive tasks such as semantic segmentation, instance segmentation, object detection, etc.

[0039] 4.The present application utilizes the characteristics of coarse-grained features providing more semantic level features and fine-grained features providing more local detail level features, and realizes the fusion of semantic level features into fine-grained features through an attention mechanism, thereby strengthening the effective semantic object areas in fine-grained features and suppressing the background areas. The finally fused features can be used in pixel-intensive tasks such as semantic segmentation and object detection, thereby improving the overall performance of these tasks, especially the segmentation or detection of small scale targets and objects with low boundary differentiation. The combination of different granularity features provides a new idea. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a first fusion module (fusion module between global granularity features and block granularity features) diagram of an embodiment of the present application.

[0041] Figure 2 is a second fusion module (fusion module between coarse-grained features and fine-grained features) diagram of an embodiment of the present application.

[0042] Figure 3 is a network structure diagram of the first fusion module and the second fusion module embedded in the coding and decoding semantic segmentation architecture of an embodiment of the present application. DETAILED DESCRIPTION

[0043] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0044] Referring to Figure 3 , the embodiments of the application include two fusion modules for different granularity features, i.e., a first fusion module and a second fusion module; the first fusion module is used to realize fusion between block granularity features and global features; and the second fusion module is used to realize fusion between global block granularity features and fine granularity features.

[0045] I. The first fusion module realizes fusion between block granularity features and global features:

[0046] The features obtained through convolution operation are input into a windowed multi-layer perceptron W-MLP to obtain block granularity features, denoted as f m ; the features obtained through convolution operation are input into a multi-layer perceptron MLP to obtain global granularity features, denoted as f g ; and the extracted block granularity features f m and global granularity features f g are respectively input into the first fusion module (see Figure 1 ).

[0047] 1-1) The block granularity feature map f m is extracted through a windowed Swin Transformer network (or a windowed multi-layer perceptron W-MLP), and the global granularity feature map f g is extracted through a Transformer network (or a multi-layer perceptron MLP), each type of feature having a size of 2Hx2WxC or HxWxC'. The first fusion module sequentially performs 1x1 convolution and Sigmoid operation on the global granularity feature map to obtain attention weight coefficients of each position in the global feature map. The output feature is subjected to dimension transformation operation to obtain a global attention weight factor feature map f w with semantic information, having a feature size of HxWx1xC; and this process is represented by the following formula (1):

[0048] f w = Reshape (Sigmoid (Conv1x1 (f g ))) (1)

[0049] In the above formula, Conv1x1 and Reshape represent 1x1 convolution and feature map dimension transformation.

[0050] 1-2) The block granularity feature map f mPixel Unshuffle operation is performed on the feature map space resolution scale of 2Hx2WxC, which is reduced to 1 / 4 of the original, and the channel dimension is increased to 4 times of the original. Through the dimension transformation operation, the output feature scale size is HxWx4xC. This process is represented by the following formula (2):

[0051] f m =Reshape(Pus(f m )) (2)

[0052] The parameters Pus and Reshape in the above formula represent the Pixel Unshuffle and feature map dimension transformation operations, respectively.

[0053] 1-3) The global attention weight factor feature map f w obtained in 1-1 is multiplied by the block granularity feature map f m obtained in 1-2; the multiplied fusion feature is added to the global attention weight factor feature map f w to strengthen the global semantic feature; finally, the Pixel Shuffle operation and the channel transformation operation are performed to obtain the fusion feature map f b after the global granularity feature strengthening, and the feature scale size is 2Hx2WxC.

[0054] f b =Reshape(Ps(f m ×fw+f w )) (3)

[0055] The parameter P s in the above formula represents the Pixel shuffle operation.

[0056] 1-4) The global granularity feature map f g is subjected to two times of bilinear interpolation, and then the interpolated output feature map is subjected to layer normalization operation; at the same time, the fusion feature map f b after the global granularity feature strengthening output by 1-3 is subjected to layer normalization operation; finally, the two types of features output by the layer normalization operation are merged together through concatenation operation, and the block granularity and global granularity fusion feature f u , i.e., the coarse granularity feature map, is finally output. The operation of this step is represented by the following formula:

[0057] f u =Cat(LN(Up(f g )),LN(f b )) (4)。

[0058] where LN, Up and Cat represent layer normalization, bilinear up-sampling and feature concatenation operation respectively.

[0059] Second, the second fusion module realizes the fusion of the block granularity and global granularity fusion features and the fine-grained features:

[0060] The coarse-grained feature map output by step one and the fine-grained feature map obtained by convolution operation are taken as the input of the second fusion module (see Figure 2 ). The second fusion module realizes the fusion of fine-grained features and block granularity and global granularity features. The output of the fusion is taken as the input feature of the subsequent segmentation module.

[0061] Let f u , f aw , f c , f o and f a represent the block granularity and global granularity fusion features (obtained by the first fusion module), the attention weight map, the fine-grained feature, the edge enhancement feature map and the final output fusion feature map respectively. The objects processed by the second fusion module are the fine-grained features of local details and the coarse-grained features fused with global and block granularity respectively. The designed module two realizes the enhancement of important local details (such as edges) and the suppression of information irrelevant to abstract semantics. Finally, the fused features have important details and abstract semantics; the specific process of the second fusion module (see Figure 2 ) to fuse the fine-grained features output by convolution operation and the coarse-grained (local and global) features output by module one is as follows:

[0062] 2-1) Extract fine-grained features by convolutional neural network, denoted as f c ; input the extracted coarse-grained features f u and fine-grained features f c into the second fusion module at the same time. Perform 1x1 convolution operation and Sigmoid operation on the block granularity and global granularity fusion features f u output by the first fusion module (denoted as coarse-grained feature map) to extract the semantic level attention weight feature map f aw with the same channel dimension and semantic information as the fine-grained features, and the size of the feature map is denoted as 2Hx2WxC; this step is represented by the following formula:

[0063] f aw = Sigmoid(Conv1x1(f u )) (5)

[0064] In the above formula, Conv1x1 represents 1x1 convolution operation on the coarse-grained feature f aw , and outputs the normalized attention feature weight feature map f aw .

[0065] 2-2) The attention weight feature map f aw output in 2-1) is multiplied with the fine-grained feature map f c ; then the multiplied feature map is input into a Sobel edge extraction operator to perform edge extraction to obtain an edge attention feature map; finally, the fine-grained feature map f c is added to the edge attention feature map through a jump connection to obtain an attention feature output map f o after edge strengthening on a semantic level for the fine-grained feature; this step is represented as:

[0066] f o = Sobel(f aw x f c ) + f c (6).

[0067] In the above formula, Sobel represents a Sobel edge extraction operation on the attention weight map.

[0068] 2-3) The coarse-grained feature f u and the edge strengthened feature f o output in 2-2) are respectively subjected to layer normalization operations; the two types of features output after the layer normalization operations are merged through a Cat operation to output a final fused feature map f a ; this step is represented by the following formula:

[0069] f a = Cat(LN(f u ), LN(f o )) (7)

[0070] In the above formula, the parameters LN and Cat respectively represent a layer normalization operation on a feature and a feature Cat operation.

[0071] III. The module one and the module two are used in a coding and decoding semantic segmentation structure (as shown in FIG. 3) in a specific implementation process as follows: Figure 3

[0072] In the uppermost layer of the encoding end, a convolution operation is used to extract a fine-grained feature, a windowed MLP is used to extract a block-grained feature, and an MLP is used to extract a global feature in space; the three different features are respectively input into corresponding grained fusion modules in Figure 3 to obtain fused features; then the fused features output by each fusion module are input into a third-row decoding network; finally, a corresponding semantic segmentation result is output. Figure 3

[0073] ​​The above examples are only used to illustrate the design idea and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and the protection scope of the present application is not limited to the above examples. Therefore, any equivalent changes or modifications made according to the principles and design ideas disclosed by the present application are within the protection scope of the present application.

Claims

1. A novel multi-granularity feature fusion method based on attention mechanism, characterized in that: Comprising the following steps: S0: building a new multi-granularity feature fusion system based on attention mechanism, including an encoding network, a decoding network, a first fusion module and a second fusion module; The first fusion module is used for fusing the global granularity feature map and the block granularity feature map, and outputs a coarse-granularity feature map; The second fusion module is used for fusing the coarse-granularity feature map and the fine-granularity feature map, and outputs a final fusion feature map; S1: input the image into the encoding network, use convolution operation to extract the fine-grained feature map f of the image c , the feature scale is 2Hx2WxC; S2: input the features obtained through convolution operation into a windowed multi-layer perceptron W-MLP to extract a block granularity feature map f of the image m , and the feature scale is 2Hx2WxC; S3: input the feature obtained through the convolution operation into a multi-layer perceptron (MLP) to extract a global granularity feature map f of the image g ; the feature scale of the global granularity feature map f g is HxWxC'. S4: the global granularity feature map f g and the block granularity feature map f m input a first fusion module; let f w represent a global attention weight factor feature map, and the feature scale is HxWx1xC; let f b represent a fusion feature map after the global granularity feature is strengthened, and the feature scale is 2Hx2WxC; let f u represent a coarse granularity feature map, and the fusion process of the first fusion module is represented by the following formula: f w = Reshape(Sigmoid(Conv1x1(f g ))) (1), f m = Reshape(Pus(f m )) (2), f b = Reshape(Ps(f m ×f m ×f w + fw)) (3), f u = Cat(LN(Up(f g )), LN(f b )) (4), In the above formula, Conv1x1 represents a 1x1 convolution, Sigmoid represents a Sigmoid operation, Reshape represents a feature map dimension transformation, Pus represents a Pixel Unshuffle pixel reorganization operation, Ps represents a Pixel shuffle operation, LN represents a layer normalization operation, Up represents a bilinear upsampling operation, and Cat represents a feature concatenation operation; S5: input the coarse-grained feature map f u with the fine-grained feature map f c into the second fusion module; let f aw represent the attention weight feature map, and the feature scale is 2Hx2WxC; let f o represent the edge enhancement feature map; let f a represent the final fusion feature map; and the fusion process of the second fusion module is represented by the following formula: f aw = Sigmoid(Conv1x1(f u )) (5), f o = Sobel(f aw × f c ) + f c (6), f a = Cat(LN(f u ), LN(f o )) (7), In the above formula, Sobel represents a Sobel edge extraction operation; S6: inputting the feature maps fused by the first fusion module and the second fusion module at each step into the decoding network, and outputting corresponding semantic segmentation results.

2. The novel multi-granularity feature fusion method based on attention mechanism according to claim 1, characterized in that: In step S4, the first fusion module fuses the global granularity feature map f g and the block granularity feature map f m The specific steps are as follows: S41: The first fusion module performs 1x1 convolution and a Sigmoid operation on the global feature map f g S41: The first fusion module performs 1x1 convolution and a Sigmoid operation on the global feature map f g S41: The first fusion module performs 1x1 convolution and a Sigmoid operation on the global feature map f w ; S42: a block granularity feature map f m Pixel Unshuffle pixel reorganization operation is performed to reduce the feature map spatial resolution scale by 1 / 4 and increase the channel dimension by 4 times; through dimension transformation operation, a block granularity feature map f m The scale of the block granularity feature map f is HxWx4xC. S43: multiply the global attention weight factor feature map f w with the block granularity feature map f m after step S42 operation, and then add the multiplied result with the global attention weight factor feature map f w to strengthen the global semantic features; finally, after the Pixel Shuffle operation and the channel conversion operation, the fusion feature map f b after the global granularity feature strengthening is obtained. S44: the global granularity feature map f g The bilinear interpolation is performed twice, and then the layer normalization operation is performed on the interpolated output feature map; the fusion feature map f b The layer normalization operation is performed; the two types of features output by the layer normalization operation are merged through the parallel connection operation, and the coarse granularity feature map f u .

3. The novel multi-granularity feature fusion method based on attention mechanism according to claim 2, characterized in that: In step S5, the second fusion module fuses the coarse-grained feature map f u with the fine-grained feature map f c The specific steps are as follows: S51: the second fusion module performs a 1*1 convolution operation on the coarse-grained feature map f u The 1*1 convolution operation and the Sigmoid operation are sequentially performed to obtain an attention weight feature map f with the same channel dimension as the fine-grained one and semantic information aw ; S52: multiply the attention weight feature map f aw outputted by step S51 with the fine-grained feature map f c ; then input the multiplied result into a Sobel edge extraction operator to extract an edge attention feature map; finally, add the fine-grained feature map f c and the edge attention feature map through a jump connection to obtain an edge enhancement feature map f o that edge-enhances the fine-grained feature on a semantic level; S53: performing a layer normalization operation on the coarse-grained feature map f u and the edge-enhanced feature map f o respectively; and merging the two types of features outputted by the layer normalization operation through a concatenation operation to output a final fused feature map f a .

4. The novel multi-granularity feature fusion method based on attention mechanism according to claim 1, characterized in that: In the step S4, the "windowed multilayer perceptron W-MLP" is replaced with "windowed Swin Transformer network".

5. The novel multi-granularity feature fusion method based on attention mechanism according to claim 1, characterized in that: In the step S5, the "multilayer perceptron MLP" is replaced with "Transformer network".

6. A computer storage medium, characterized in that: The computer program stored therein can be executed by a computer processor, and the computer program executes a new multi-granularity feature fusion method based on attention mechanism according to any one of claims 1-5.

Images