A method of joint multi-dimensional visual attention and medium and device thereof

Abstract

The application discloses a kind of joint multi-dimensional visual attention method and its medium and equipment, it is related to the field of deep learning, the present technical problem of extraction of model feature weight is extracted in visual attention method in this scheme.In attention module, respectively, two dimensions and channel dimension of space are combined, and spatial attention is embedded in channel attention, to improve the interactivity of information in different dimensions.And through the form of global, obtain spatial attention, enhance the modeling of spatial position relationship.It is characterized in that, the information interaction is effectively enhanced by the joint of multi-dimension, and by dimension compression, efficiently obtain global information of space, reduce the computing overhead.The attention method can be flexibly applied to the classical convolutional neural network, improve the extraction ability of model features, and then improve the image classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and in particular to a joint multi-dimensional visual attention method, medium and device thereof. BACKGROUND

[0002] In recent years, attention mechanism has attracted widespread attention and has been applied to various fields of computer vision. Attention mechanism is used to improve the perception of objects of interest and reduce the weight of objects of no interest. It is widely studied and applied to feature modeling of convolutional neural networks to improve the performance of modern deep neural networks.

[0003] Squeeze-and-excitation [1] is an early channel attention mechanism that extracts channel features after global average pooling in the spatial dimension to improve network performance at very low computational cost, but ignores the importance of location information. Some other works [2][3] add spatial attention to channel attention and use convolution technology to capture spatial local relationships, but cannot model long-term dependencies. Joint attention [4] extracts fine-grained channel attention at different spatial positions, but ignores modeling of spatial position relationships. These methods mainly study how to effectively extract information in the channel dimension or spatial dimension, and do not make good use of the interaction information in different dimensions to efficiently combine spatial attention and channel attention.

[0004] The self-attention mechanism in visual Transformer [5] can model global relationships in space and improve the network's global scene understanding ability, effectively making up for the narrowness and locality of convolution technology. Although the self-attention mechanism performs outstandingly in many visual tasks, the large number of parameters and calculations of the self-attention mechanism makes it impossible to be applied to small convolutional neural networks in a lightweight and flexible manner.

[0005] [1] Hu J, Shen L, Sun G. Squeeze-and-excitation networks [A]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition [C]. 2018: 7132-7141.

[0006] [2] Park J, Woo S, Lee J Y, et al. Bam: Bottleneck attention module [J]. arXiv preprint arXiv:1807.06514, 2018.

[0007] [3] Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module[A]. Proceedings of the European Conference on Computer Vision[C]. 2018: 3-19.

[0008] [4] Hou Q, Zhou D, Feng J. Coordinate attention for efficient mobile network design[A]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition[C]. 2021: 13713-13722.

[0009] [5] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020. SUMMARY

[0010] In order to at least solve one of the technical problems existing in the prior art, the present application provides a joint multi-dimensional visual attention method and its medium and device, which proposes a joint multi-dimensional visual attention method, embeds a light global attention while obtaining interactive information of different dimensions, can greatly reduce the huge computing overhead brought by self-attention mechanism, make up for the deficiency of convolution operation in obtaining attention information, effectively combine spatial attention and channel attention, and make the attention module light and efficient.

[0011] In order to achieve the purpose of the present application, the present application provides a joint multi-dimensional visual attention method, which comprises the following steps:

[0012] S1, input image data and determine the channel compression rate;

[0013] S2, establish a convolutional neural network model, add a visual attention module to the end of each convolutional combination block in the convolutional neural network, separate the spatial dimension of the feature map in the joint multi-dimensional visual attention module, respectively compress and aggregate the channels in proportion, encode spatial information, extract global features, and then expand and restore each dimension of the attention map;

[0014] S3, reweighting the extracted multi-dimensional attention map along the different spatial dimensions of the input feature map, iteratively training the model using gradient descent method to obtain an image classification model for classifying images, and inputting the image to be classified into the image classification model, so as to classify the image.

[0015] Further, the channel compression rate in step S1 is determined according to the parameter amount target of the model.

[0016] Further, in step S2, the spatial dimensions of the separated feature maps are separated in the joint multi-dimensional visual attention module.

[0017] Given input C represents the channel dimension, and H and W represent the two spatial dimensions. Global average pooling is used on the two spatial dimensions H and W to aggregate spatial features along the horizontal and vertical coordinates respectively, generating a pair of feature maps with direction perception ability. Therefore, the output of the hth row and wth column on the cth channel can be represented as:

[0018]

[0019]

[0020] In step S2, the channel features are proportionally compressed and aggregated;

[0021] Through the joint of spatial dimensions and channel dimensions, the feature map contains interaction information of space and channel. At this time, the feature map is further compressed in the channel dimension. In order to obtain the channel attention corresponding to different spatial positions, the feature maps along the H dimension and the W dimension are connected in the channel dimension, and a shared 1x1 convolution transformation function F1 is used to aggregate channel features.

[0022] f = F1 ([z H , z W ])

[0023] Where [·, ·] represents the connection operation along the spatial dimension, r represents the intermediate feature map after aggregating channel features in the horizontal and vertical directions. r c is the reduction ratio of the channel when aggregating. Then f is divided into two independent matrices and

[0024] Step S3 encodes spatial information.

[0025] The self-attention module models the global dependence of the feature map f H and f W , and shares the weights of the information extracted in different dimensions.

[0026] p H = δ(SA(f H ))

[0027] p W = δ(SA(f W ))

[0028] SA represents a self-attention function, and δ represents a ReLU activation function. The self-attention mechanism inputs a matrix with a sequence length of L and a channel length of D The input matrix X' is linearly transformed by three fully connected layers with weights (W Q , W K , W V ) respectively to obtain matrices (Q, K, V). T After matrix multiplication and division by the channel dimension , the Softmax function is used for normalization, and finally the value matrix V is multiplied to obtain the output SA(X').

[0029] (Q, K, V) = (X'W Q , X'W K , X'W V )

[0030]

[0031] Step S2 restores the feature dimension of the attention map.

[0032] After spatial attention, the feature maps p H and p W are obtained.

[0033] g = σ(F2([p H , p W ])) where [·, ·] represents a connection operation along the spatial dimension, and σ is a Sigmoid activation function, represents the feature map after restoring the channel size in the horizontal and vertical directions. and two attention maps.

[0034] The joint multi-dimensional visual attention method and system of the present application use the visual attention method described above to extract feature weights of a convolutional neural network.

[0035] The joint multi-dimensional visual attention method and system have the advantages that the multi-dimensional joint effectively enhances information interaction, and through dimension compression, global information of space is efficiently obtained, and calculation overhead is reduced.

[0036] The spatial dimension of the input feature map is separated, the spatial dimension and the channel dimension are combined, the spatial attention is embedded into the channel attention, and the information interaction between multiple dimensions is improved. Meanwhile, through compression of the spatial and channel features, the global relationship of the spatial position is modeled by combining the self-attention mechanism, and the calculation overhead of the self-attention module is greatly reduced. The attention module can be easily added to the classic convolutional neural network. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a schematic diagram of the module integrated with the convolutional neural network ResNet.

[0038] Figure 2 is a flowchart of the joint multi-dimensional visual attention method in the embodiment of the application.

[0039] Figure 3 is a schematic diagram of the global information extraction process. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some but not all of the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the protection scope of the application.

[0041] The channel and spatial attention can enhance the feature extraction capability of the model, but the simple sequential combination can lead to low efficiency of extracted features. Therefore, the application adopts the joint dimension method, separates and compresses the spatial dimension of the input feature map, combines the channel dimension, embeds the spatial attention into the channel attention, models the global relationship of the spatial position by combining the self-attention mechanism, adds the attention module to the convolutional neural network for simultaneous training, and finally obtains a convolutional neural network with efficient attention. Please refer to Figures 1-3 As shown in the figure, the joint multi-dimensional lightweight visual attention method provided by the application includes the following steps:

[0042] S1, constructing a data set and determining a channel compression rate.

[0043] In some embodiments of the present application, the disclosed image classification dataset is divided into a training set and a test set, and the images in the dataset are preprocessed such as padding, cropping, flipping and normalization. Among them, the training set is used to train the weight and structure parameter of the model, and the test set is used to test the effect of the visual attention method. At the same time, according to the parameter amount target setting of the added channel compression rate in the visual attention method, the smaller the compression rate, the larger the parameter amount, and the higher the precision.

[0044] S2, a convolutional neural network model is established, and a visual attention module is added to the end of each convolutional combination block in the convolutional neural network.

[0045] In some embodiments of the present application, as shown in Figure 1 , taking the classic convolutional neural network ResNet as an example, a visual attention module is added to the end of each basic block or bottleneck block.

[0046] S3, in the joint multi-dimensional visual attention module, the spatial dimensions of the feature map are separated, and the channels are compressed and aggregated in proportion respectively, the spatial information is encoded, the global features are extracted, and then each dimension of the attention map is expanded and restored. The specific structure is as shown in Figure 2 、 3 .

[0047] Specifically:

[0048] Joint multi-dimensional, aggregate spatial features. Given the input where C represents the channel dimension, H and W represent the width and height of the two spatial dimensions respectively, represent the real field. Global average pooling is used for the two spatial dimensions H and W, and the spatial features are aggregated along the horizontal and vertical coordinates respectively to generate a pair of feature maps with direction perception ability. Therefore, the output feature map of the cth channel on the hth row and the wth column can be represented as and

[0049]

[0050]

[0051] where, x c (h, j) represents the value of the cth channel on the hth row and the jth column in the input feature map X, x c (i, w) represents the value of the cth channel on the ith row and the wth column in the input feature map X.

[0052] Aggregation of channel features. Through the joint of spatial dimension and channel dimension, the interaction information of space and channel is contained in the feature map, and the feature map is further compressed in the channel dimension. Specifically, in order to obtain the channel attention corresponding to different spatial positions, the feature maps along the W dimension and the H dimension are connected in the channel dimension, and a shared 1x1 convolution transformation function F1 is used to aggregate the channel features, where [·, ·] represents the splicing operation along the spatial dimension, represents the intermediate feature map after aggregating the channel features in the horizontal direction and the vertical direction. c is the reduction ratio of the channel when aggregating. Then f is split into two independent matrices along the spatial dimension and

[0053] Encoding spatial information. The feature map f H is encoded by a self-attention module for extracting global information W Modeling the global dependence of space, the weights for extracting information in different dimensions are shared, which improves the generalization of the model and reduces the parameter amount of the model. After f H and f W are encoded for spatial information, output feature maps p H and p W are obtained respectively:

[0054] p H = δ(SA(f H ))

[0055] p W = δ(SA(f W ))

[0056] SA represents a self-attention function, and δ represents a ReLU activation function.

[0057] The self-attention module inputs a matrix X' with a sequence length L and a channel length D SA projects X' to a query matrix Q, a key matrix K and a value matrix V through linear operation. In the network, the spatial dimension W of fW corresponds to the input sequence length L of the self-attention mechanism, and the channel dimension C / r c of f W corresponds to the input channel length D of the self-attention mechanism. In some embodiments of the present application, for the SA self-attention mechanism of the W dimension as shown in Figure 3 , the input matrix X' is linearly transformed by three fully connected layers with weights (W Q , W K , W V ) respectively to obtain matrices (Q, K, V). The query matrix Q and the transposed (T) key matrix K T are multiplied by matrices, and divided by the channel dimension After normalization by using Softmax function, the output SA(X') is obtained by matrix multiplication with the value matrix V.

[0058] (Q, K, V) = (X'W Q , X'W K , X'W V )

[0059]

[0060] Feature dimension reduction. After spatial attention, the feature maps p H and p W are obtained. The feature maps are concatenated and expanded by a shared 1x1 convolution transformation function F2, to obtain

[0061] g = σ(F2([p H , p W ]))

[0062] where [·, ·] represents the concatenation operation along the spatial dimension, σ is the Sigmoid activation function, represents the feature maps after reducing the channel size in the horizontal and vertical directions. Then g is divided into two independent matrices along the spatial dimension, to obtain and two attention maps.

[0063] S4, after the attention extraction is completed, the two attention maps are weighted along the two directions of the space to the input feature map X, to obtain the output feature map Y.

[0064] y(i, j) = x(i, j) x g H (i) x g W (j)

[0065] where y(i, j) represents the channel of the i-th row and the j-th column in the output feature map Y, x(i, j) represents the channel of the i-th row and the j-th column in the input feature map X. g H (i) represents the i-th row of the attention map along the spatial height direction, and g w (j) represents the j-th column of the attention map along the spatial width direction.

[0066] After the feature map Y is reweighted, the forward propagation of the model is continued, and the model parameters are updated by back propagation. After training the model using gradient descent method, a general and efficient image classification network can be obtained. After adding the joint multi-dimensional visual attention method to the benchmark model, the image classification accuracy can be greatly improved with only a small increase in parameters.

[0067] The application further provides a storage medium, which can be a ROM, a RAM, a magnetic disk, an optical disk or the like storage medium, and the storage medium stores one or more programs, and the program is executed by a processor to realize the joint multi-dimensional visual attention method provided in the above embodiment.

[0068] The application further provides a device, which can be a desktop computer, a notebook computer, a smart phone, a PDA handheld terminal, a tablet computer or other terminal device with a display function, and the computing device comprises a processor and a memory, the memory stores one or more programs, and the processor executes the program stored in the memory to realize the joint multi-dimensional visual attention method provided in the above embodiment.

[0069] The application can optimize feature extraction of a convolutional neural network by using the visual attention method.

[0070] The application proposes a joint spatial and channel combined attention method, which embeds spatial feature extraction when capturing channel attention of different spatial positions, and efficiently utilizes the interaction information between the spatial and the channel.

[0071] (2) The existing spatial and channel combined attention module adopts a spatial local attention extraction mode to obtain spatial attention information, and the application adopts a self-attention mechanism to capture long-distance dependence of the spatial, thereby improving the performance of the attention mechanism.

[0072] (3) The self-attention mechanism can extract global features of the spatial, but will introduce a large amount of calculation. The application uses the self-attention mechanism after squeezing the channel and spatial features, thereby extracting global features of the spatial while greatly reducing the calculation amount of the model.

[0073] (4) In the compression and expansion process of the channel features, the application shares convolution parameters, thereby improving the generalization ability of the model and greatly reducing the parameter amount of the attention module. The application can be easily inserted into a classical deep neural network.

[0074] The image is classified by using the model, and the image classification precision can be greatly improved.

[0075] For those skilled in the art, various corresponding changes and deformations can be made according to the above-described technical solutions and concepts, and all these changes and deformations should belong to the protection scope of the claims of the application.

Claims

1. A method of joint multi-dimensional visual attention, the method comprising: The method comprises the following steps: S1, constructing a data set and determining a channel compression rate; S2, establishing a convolutional neural network model, adding a visual attention module to the end of each convolutional combination block in the convolutional neural network, wherein in the joint multi-dimensional visual attention module, the spatial dimensions of the feature map are separated, the channels are compressed and aggregated respectively, the spatial information is encoded, the global features are extracted, then the dimensions of the attention map are expanded and restored to obtain two attention maps; S3, reweighting the extracted multi-dimensional attention map along the different spatial dimensions of the input feature map, and iteratively training the model based on the data set using the gradient descent method to obtain an image classification model for classifying images, and inputting the image to be classified into the image classification model to classify the image; The specific operation of the joint multi-dimensional visual attention module in step S2 comprises: Joint multi-dimension, aggregate spatial features: given input where represents the channel dimension, , represents the two dimensions of space, represents the real field, for two spatial dimensions and using global average pooling, respectively along the horizontal and vertical coordinates to aggregate spatial features, generating a pair of feature maps with direction perception ability; aggregate channel features: concatenating along the channel dimension dimension and dimensional feature maps, and adopt shared convolution transformation functions aggregate channel features, to obtain intermediate feature maps after aggregating channel features in the horizontal direction and the vertical direction , then split along the spatial dimension into two independent matrices and , is the reduction ratio of the channel when aggregating; Encoding spatial information: through the self-attention module to the feature map And Modeling the global dependence of the space, sharing the weights of the information extracted in different dimensions, and And After encoding the spatial information, output feature maps are obtained respectively And ; Feature dimension reduction: feature maps after spatial attention and After that, concatenate feature maps and pass through shared convolutional transformation function dilated channel features, resulting in wherein, represents a concatenation operation along the spatial dimension, is a Sigmoid activation function, represents the feature maps after restoring the channel size in the horizontal and vertical directions, and then concatenating into two independent matrices to obtain and two attention maps; output feature map and the expression for denotes a self-attention function, denotes a ReLU activation function; In the self-attention module, the input matrix is linearly transformed by three fully connected layers with weights to obtain matrices , respectively. The query matrix and the transposed key matrix are multiplied by the matrices , and then normalized by the function. Finally, the output is obtained by matrix multiplication with the value matrix : The superscript T represents transposition.

2. The method of claim 1, wherein, The channel compression rate in step S1 is determined according to the parameter amount target of the model.

3. The method of claim 1, wherein the method is a joint multi-dimensional visual attention method. The output feature maps of the first row and the first column on the first channel are respectively represented as and . The output feature maps of the second row and the second column on the second channel are respectively represented as and . wherein, denotes the value of the input feature map in the i-th channel in the i-th channel in the i-th channel in the i-th channel denotes the value of the input feature map in the i-th channel in the i-th channel in the i-th channel in the i-th channel 4. The method of claim 1, wherein, intermediate feature map wherein, represents a concatenation operation along the spatial dimension, represents the intermediate feature map after aggregating channel features in horizontal and vertical directions, is the reduction ratio of channels when aggregating.

5. The method of claim 1, wherein, In step S3, the two attention maps are respectively weighted along two directions of the space on the input feature map to obtain an output feature map . .

6. A storage medium storing one or more programs, the storage medium comprising instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method of any of claims 1-5. The program is executed by the processor to implement the joint multi-dimensional visual attention method of any one of claims 1-5.

7. A device comprising a processor and a memory, the memory storing one or more programs, the device characterized in that, The processor executes the program stored in the memory to implement the joint multi-dimensional visual attention method of any one of claims 1-5.

Images