A target feature extraction method based on cross-correlation self-attention mechanism

Abstract

The present invention relates to the technical field of target detection and recognition, and discloses a target feature extraction method based on a mutual correlation self-attention mechanism, which specifically includes first inputting a feature map Z of size H*W*C; then performing a window division operation on the feature map; then expanding the channel dimension of the linear layer to 2*C, dividing the matrix along the channel dimension into a matrix M and a matrix V; obtaining a mutual correlation matrix and an activation operation; then performing a self-attention calculation and a channel attention calculation; and finally outputting a feature map Y of size H*W*C. The present invention searches for the correlation between elements in the feature map, obtains similar features of the target, and simultaneously enables information sharing between channels to realize the selection of attention areas in the spatial dimension and the channel dimension. The present invention improves the recognition effect of the model on the information to be tested in the image, and improves the recognition accuracy of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and recognition, and in particular to a target feature extraction method based on a cross-correlation self-attention mechanism. Background Art

[0002] With the development of deep learning, image processing methods based on convolutional neural networks have gradually become mainstream. In deep learning, with the rapid growth of data and information, how to utilize attention mechanisms to focus limited computing power on target areas has become a current research hotspot. Currently, many studies are integrating attention mechanisms into feature extraction. First, convolutional neural networks can use attention mechanisms to automatically calculate feature regions that need to be highlighted using learnable weights. Second, they can mimic human attention behavior and identify focal areas in images.

[0003] Attention mechanisms can be categorized into three types: spatial attention, channel attention, and self-attention. Hou et al. proposed the Coordinate Attention (CA) mechanism, which processes feature maps in the spatial dimension and performs pooling operations in two dimensions, capturing long-range dependencies and precise locations. Wang et al. first performed a pooling operation on the feature map and then performed a one-dimensional convolution operation on the channel dimension of the feature map to capture inter-channel connections. To combine spatial and channel attention mechanisms, Woo et al. proposed the CBAM module, which first assigns weights in the channel dimension and then performs target search in the spatial dimension. The Transformer model was the first to apply the self-attention mechanism to natural language processing, and the ViT model subsequently extended it to computer vision. The self-attention mechanism can locate the target location by using connections between pixels in the image itself, while also capturing global information at once. It can be seen that the spatial and channel attention mechanisms based on convolution operations lack the global information capture characteristics of the self-attention mechanism, and the self-attention mechanism cannot interact with information between channels. Based on the advantages and disadvantages of existing attention mechanisms, this paper proposes a target feature extraction method based on the cross-correlation self-attention mechanism. Summary of the Invention

[0004] The purpose of the present invention is to provide a target feature extraction method based on the cross-correlation self-attention mechanism; it utilizes the ability of the traditional self-attention mechanism to capture global information, and realizes information interaction between channels through the channel attention mechanism. On this basis, the cross-correlation matrix is ​​used to find similar information in the feature map, and then find the attention area in the image, so as to achieve efficient and accurate target recognition.

[0005] The present invention is implemented as follows: a target feature extraction method based on a mutual correlation self-attention mechanism; specifically, the following steps are performed:

[0006] S1: First input the feature map Z of size H*W*C;

[0007] S2: Perform window division on the feature map; then divide the input tensor into groups of n pixels along the length and width directions, and the size of each window is n*n; finally, extend the three-dimensional tensor in the window from H*W*C to HW*1*C.

[0008] S3: The linear layer expands the channel dimension to 2*C, and divides the matrix into matrix M and matrix V along the channel dimension; specifically, the following steps are performed;

[0009] S 3.1 : Expand the channel dimension to 2*C through the linear layer, and divide the matrix into matrix M and matrix V along the channel dimension:

[0010] S 3.2 : Expand the input tensor through the linear layer to 2*C;

[0011] S 3.3 : The expanded tensor is divided into two parts along the channel dimension, namely the M matrix and the V matrix, where each column in the M matrix and the V matrix is ​​as shown in formula (1) and formula (2):

[0012] Formula (1)

[0013] Formula (2)

[0014] Among them, , C represents the number of channels of the matrix.

[0015] S4: Obtain the cross-correlation matrix; specifically, perform the following steps:

[0016] S 4.1 : Copy each column vector in the M matrix into H*W columns, then copy the resulting matrix of size (H*W)*(H*W) into two copies, transpose one of them, and finally subtract the copied matrix from its transposed matrix to obtain the difference between each element in the M matrix and other elements;

[0017] S 4.2 : Add the elements of each channel corresponding to the position in the M matrix, and the obtained molecular matrix is, ;

[0018] S 4.3 : Define the denominator matrix as , its expression is as follows:

[0019] Formula (3)

[0020] S 4.4 : Divide the numerator matrix by the denominator matrix to obtain the similarity matrix , the calculation formula is as follows (4);

[0021] Formula (4)

[0022] S 4.5 : The matrix Use a 1*1 convolution kernel for convolution operation.

[0023] S5: Activation operation: Activate the convolved tensor using the activation function, and define the activation function as shown in formula (5);

[0024] = Formula (5)

[0025] Where X is the input tensor, is the matrix after passing the activation function.

[0026] S6: Perform self-attention calculation; the activated matrix Perform dot product operation with matrix V; rearrange the tensor result after operation into H*W*C; then multiply the rearranged result with the channel weight obtained by the channel attention mechanism for the corresponding channel;

[0027] S7: perform channel attention calculation;

[0028] S8: Output feature map Y of size H*W*C.

[0029] Furthermore, a channel attention mechanism is added. First, the input feature map H*W*C is average-pooled to obtain a feature map of size 1*1*C. The feature map is convolved using a one-dimensional convolution kernel of size 3*1. The convolved feature value is activated by the Sigmoid function. The specific implementation formula is as follows (6):

[0030] Formula (6)

[0031] in, represents the tensor features of the input, represents the Sigmoid activation function, Represents a convolution operation of size 3*1.

[0032] Furthermore, the network model used is YOLOv5, and the running steps are as follows:

[0033] S8.1 : Obtain the data set and perform Mosaic data enhancement on the data set; send the enhanced data into the network for training;

[0034] S 8.2 : A target feature extraction method based on the mutual correlation self-attention mechanism is applied to the YOLOv5 network, replacing the last three C3 modules in the neck structure;

[0035] S 8.3 : Copy the input tensor into two copies and process them through two branches respectively;

[0036] S 8.4 : One of the branches undergoes 1*1 convolution and an improved self-attention mechanism; the other branch undergoes 1*1 convolution;

[0037] S 8.5 : The output results of the two branches are concat-operated, spliced ​​along the channel dimension, and then subjected to a 1*1 convolution operation;

[0038] S 8.6 The optimization algorithm uses stochastic gradient descent (SGD) as the optimizer, with 16 images as a training batch, an initial learning rate of 1e-2, a weight decay parameter of 5e-4, and a momentum of 0.937, for 300 epochs. In the initial stage of model training, three epochs are used for warm-up training.

[0039] S 8.7 : After training the model, predict the image and get the result.

[0040] Furthermore, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements any one of the methods described above when executed by a main controller.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] By finding the correlation between elements in the feature map, similar features of the target are obtained, and information sharing between channels is achieved, thus realizing the selection of attention areas in the spatial and channel dimensions. This invention improves the recognition effect of the model on the information to be tested in the image and improves the recognition accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is a flow chart of the model of the present invention.

[0045] Figure 2 This is a structural diagram of the present invention running in the YOLOv5 network.

[0046] Figure 3 This is a module principle diagram of the mutual correlation self-attention mechanism in the present invention.

[0047] Figure 4 It is a flowchart of the cross-correlation self-attention mechanism in the present invention.

[0048] Figure 5 This is a schematic diagram of the module principle of the channel attention operation in the present invention.

[0049] Figure 6 This is a schematic diagram of the molecular matrix in the mutual correlation self-attention mechanism of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] See also Figure 1-6 , a target feature extraction method based on the mutual correlation self-attention mechanism, the network model used is YOLOv5, and the modules of YOLOv5 used in the present invention include:

[0052] 1. Focus module: expands the three channels of the RGB image by four times to become twelve channels, and then performs a convolution operation to obtain a doubly downsampled feature map;

[0053] 2. Conv module: consists of two-dimensional convolution, batch normalization and activation function;

[0054] 3. SPP module: The input image is subjected to 5*5, 9*9, and 13*13 maximum pooling operations respectively, and then the fused features are obtained through convolution;

[0055] 4. C3 module: It contains three Conv modules and a Bottleneck structure. One branch passes through the Conv module and the Bottleneck module, and the other passes through the Conv module. The results of the two branches are concat-operated and pass through the last Conv module.

[0056] 5. Cross-correlation C3 module: Based on the C3 module, the Bottleneck structure is replaced with a cross-correlation self-attention mechanism.

[0057] In this embodiment, after combining the present invention with the YOLOv5 network, the operation steps are as follows:

[0058] S1: Obtain the data set, divide the data, and perform Mosaic data enhancement on the data set.

[0059] S2: Build the YOLOv5 network model.

[0060] S3: Apply a target feature extraction method based on the mutual correlation self-attention mechanism to the YOLOv5 network and replace the last three C3 modules in the neck structure. The steps are as follows:

[0061] S 3.1 : Copy the input tensor into two parts and process them through two branches respectively.

[0062] S 3.2 : One of the branches undergoes 1*1 convolution and an improved self-attention mechanism.

[0063] S 3.3 : The other branch undergoes 1*1 convolution.

[0064] S 3.5 : The output results of the two branches are concat-operated, spliced ​​along the channel dimension, and then subjected to a 1*1 convolution operation.

[0065] S4: Send the processed data set into the model for training, and detect the model results through test images.

[0066] S5: The optimization algorithm used stochastic gradient descent (SGD) as the optimizer, with a training batch size of 16 images, an initial learning rate of 1e-2, a weight decay parameter of 5e-4, and a momentum of 0.937, for 300 epochs. During the initial training phase, a warm-up training phase of 3 epochs was performed. This paper was built using the PyTorch framework, using an Intel Xeon Gold 5320 CPU @ 2.20GHz, an NVIDIA RTX A4000 GPU, and Ubuntu 18.04.

[0067] S6: The original YOLOv5 model and the YOLOv5 model with the method proposed in the present invention are compared to compare the two network test results. The results are shown in Table 1:

[0068] Table 1 Comparison of experimental results

[0069] Precision (%) Recall (%) AP (%) YOLOv5 78.6 71.2 73.3 Proposed method 79.2 76.9 77.1

[0070] Precision indicates the percentage of targets predicted by the model that are actually targets. Recall indicates the percentage of targets successfully predicted by the model. Average Precision balances these two metrics. Using recall as the horizontal axis and precision as the vertical axis, the area under the curve (PR-curve) formed by these two parameters is calculated.

[0071] Completing the above steps can achieve efficient and accurate target recognition and improve the accuracy of target prediction.

[0072] In this embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a main controller, the method described above is implemented.

[0073] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A target feature extraction method based on the cross-correlation self-attention mechanism, characterized in that: Follow these steps: S1: First input the feature map Z of size H*W*C; S2: Perform window division operation on the feature map; specifically, perform the following steps: S 2.1 : Divide the input tensor into a group of n pixels along the length and width, and the size of each window is n*n; S 2.2 : Extend the three-dimensional tensor in the window from H*W*C to HW*1*C; S3: The linear layer expands the channel dimension to 2*C, and divides the matrix into matrix M and matrix V along the channel dimension; The expanded tensor is divided into two parts along the channel dimension, where each column in the M matrix and the V matrix is ​​as shown in Equation (1) and Equation (2): Formula (1) Formula (2) in, , C represents the number of channels of the matrix; S4: Obtain the cross-correlation matrix; specifically, perform the following steps: S 4.1 : Copy each column vector in the M matrix into H*W columns, then copy the resulting matrix of size (H*W)*(H*W) into two copies, and transpose one of them. Finally, subtract the copied matrix from its transposed matrix to obtain the difference of each element in the M matrix, which is defined as M dis ; S 4.2 :M dis The molecular matrix obtained by adding the elements of the corresponding position of each channel in the matrix is ; S 4.3 : Define the denominator matrix as , its expression is as follows: Formula (3) S 4.4 : Divide the numerator matrix by the denominator matrix to obtain the similarity matrix , the calculation formula is as follows (4); Formula (4) S 4.5 : The matrix Use a 1*1 convolution kernel for convolution operation; S5: Activation operation; specifically, the following steps are performed: the convolved tensor is activated using an activation function, and the activation function is defined as in formula (5); = Formula (5) in, X is the input tensor, is the matrix after passing the activation function; S6: Perform self-attention calculation; specifically, perform the following steps: Perform dot product operation with matrix V; rearrange the tensor result after operation into H*W*C; then multiply the rearranged result with the channel weight obtained by the channel attention mechanism for the corresponding channel; S7: perform channel attention calculation; S8: Specifically, by running the YOLOv5 network model, the running steps are as follows: S 8.1 : Obtain the data set and perform Mosaic data enhancement on the data set; send the enhanced data into the network for training; S9: The optimization algorithm uses the stochastic gradient descent algorithm SGD as the optimizer, with 16 images as a training batch, the initial learning rate of the model is 1e-2, the weight decay parameter is 5e-4, and the momentum is 0.

937. The training is done for 300 epochs. In the initial stage of model training, 3 epochs are used for warm-up training; S 10 : After training the model, predict the image and get the result.

2. The target feature extraction method based on the cross-correlation self-attention mechanism according to claim 1 is characterized in that: Add a channel attention mechanism. First, the input feature map H*W*C is average pooled to obtain a feature map of size 1*1*C. The feature map is convolved with a one-dimensional convolution kernel of size 3*1; the convolved feature value is activated by the Sigmoid function, and its specific implementation formula is as follows (6): Formula (6) in, represents the tensor features of the input, represents the Sigmoid activation function, Represents a convolution operation of size 3*1.

3. The target feature extraction method based on the cross-correlation self-attention mechanism according to claim 1 is characterized in that: In step S8.1, the following steps are specifically included: S 8.2 : A target feature extraction method based on the mutual correlation self-attention mechanism is applied to the YOLOv5 network, replacing the last three C3 modules in the neck structure; S 8.2.1 : Copy the input tensor into two copies and process them through two branches respectively; S 8.2.2 : One of the branches undergoes 1*1 convolution and an improved self-attention mechanism; the other branch undergoes 1*1 convolution; S 8.2.3 : The output results of the two branches are concat-operated, spliced ​​along the channel dimension, and then subjected to a 1*1 convolution operation.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the main controller, the method according to any one of claims 1 to 3 is implemented.

Images