A sparse attention target detection method based on improved DETR model

Abstract

The present invention discloses a sparse attention target detection method based on an improved DETR model, which is improved based on the Deformable DETR framework. The encoder is composed of a plurality of encoder layers, each layer mainly consisting of an instance-dependent sparse attention module, a gated linear control unit, and a residual connection and normalization operation between them. The decoder is composed of a plurality of decoder layers, each layer mainly consisting of a multi-head self-attention module, an instance-dependent sparse attention module, a gated linear control unit, and a residual connection and normalization operation between them. The present invention utilizes the dependency between instances to enhance the expression ability of attention. The sparse attention can dynamically adjust the connectivity between features according to the content of the input image, better capture semantic information, and reduce computational complexity. The present invention can improve computational efficiency and improve the model's detection performance for targets.

Description

Technical Field

[0001] The present invention relates to a target detection method, and in particular to a sparse attention target detection method based on an improved DETR model. Background Art

[0002] Object detection aims to accurately identify and locate specific objects in images or videos and is an important research direction in computer vision. For many years, convolutional neural networks (CNNs) have been the mainstay of object detection models. However, the tremendous success of the Transformer in natural language processing (NLP) has prompted researchers to explore its potential in computer vision (CV). The Transformer architecture has been shown to be effective in capturing long-range dependencies in sequential data.

[0003] In the field of object detection, the core model based on the Transformer architecture is DETR (Detection Transformers). The DETR model uses the Transformer encoder and decoder structure to handle object detection. The encoder is used to extract features from the input image, while the decoder is used to generate object predictions at each location. The core idea of ​​DETR is to transform the object detection problem into a collective prediction problem, namely predicting the object category, bounding box coordinates, and whether the object exists at each location, avoiding the need for traditional anchor box methods. Among improved models based on DETR, many models have adopted the multi-scale feature fusion and deformable attention concepts of Deformable DETR. Deformable DETR proposes a new attention mechanism, Deformable Attention, to process image feature maps and improve model performance. This introduction has been widely used in improved versions of DETR to address DETR's problems of slow training convergence and limited feature space resolution.

[0004] The deformable attention mechanism in Deformable DETR is indeed a good innovative method. It only focuses on a few sampling points to perform attention calculations, thereby reducing computational complexity. However, this method also brings some disadvantages and limitations: First, deformable attention performs intensive deformable operations on the entire feature map, requiring additional offset predictions for the sampling points, which leads to higher computational costs. Second, it may lead to information loss. Although the deformable attention mechanism can reduce computational complexity, if the sampling points of attention are insufficient or do not well cover the target or key information area, ignoring other important information will lead to a decline in target detection or recognition performance. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a sparse attention target detection method based on the improved DETR model with high computational efficiency and good detection performance.

[0006] Technical solution: The sparse attention target detection method based on the improved DETR model described in the present invention includes:

[0007] (1) Input the training data set into the backbone network Swin Transformer V1 and extract the three-layer feature maps C3, C4 and C5;

[0008] (2) Through the multi-scale feature fusion module, the three-layer feature maps C3, C4 and C5 are transformed into four feature layers, the four feature layers are fused and the relative position encoding information is added to obtain a multi-scale fused feature map;

[0009] (3) The multi-scale fusion feature map is used as the input of the encoder. The encoder is composed of multiple encoder layers. Each layer is mainly composed of an instance-dependent sparse attention module, a gated linear control unit, and a residual connection and normalization operation between them. The input feature sequence is processed in turn by the instance-dependent sparse attention module, the residual connection and normalization operation, and the gated linear control unit. Finally, it is processed by a residual connection and normalization operation to obtain the output of a layer of encoder. Repeat the encoder multiple times to obtain the encoder output feature map.

[0010] (4) The encoder output feature map is used as the input of the decoder. The decoder is composed of multiple decoder layers stacked together. Each layer is mainly composed of a multi-head self-attention module, an instance-dependent sparse attention module, a gated linear control unit, and residual connections and normalization operations between them. The feature sequence with position encoding is input into the multi-head self-attention module. The output of the multi-head self-attention module is processed by residual connection and normalization operations, and then processed by instance-dependent sparse attention module, residual connection and normalization operations and gated linear control unit in sequence. Finally, it is processed by a residual connection and normalization operation to obtain the output of a layer of decoder. Repeat the traversal of the decoder multiple times to obtain the decoder output feature vector.

[0011] (5) The decoder output feature vector is passed through a linear layer and a multi-layer perceptron to predict the category and bounding box, respectively, to obtain a set of predicted targets, each of which contains category and bounding box coordinate information;

[0012] (6) Calculate the overall network loss between the predicted target set and the real target set, and optimize the model through back propagation;

[0013] (7) Repeat steps (1) to (6) multiple times to obtain a trained target detection model.

[0014] Furthermore, step (1) includes:

[0015] The original input feature map size is H×W×3, where H represents the height of the image and W represents the width of the image;

[0016] The backbone network Swin Transformer V1 is used to extract three layers of feature maps C3, C4, and C5, with sizes as follows: and

[0017] Furthermore, step (2) includes:

[0018] The three-layer feature maps C3, C4 and C5 are transformed into the size of and The feature map of the last layer, C5, is transformed into a size of The feature map of , as the fourth feature layer;

[0019] The four feature layers are added with coordinate information. In order to distinguish the position information of feature points in different layers, relative position coordinates are introduced. The position embedding method is to convert the absolute coordinates of the feature points of each layer into relative coordinates; the relative coordinates and scale information of the feature points of each layer are spliced ​​together to obtain a multi-scale fusion feature map.

[0020] Furthermore, relative position encoding includes learnable scale-level embedding and position embedding.

[0021] Furthermore, in step (3), the instance-dependent sparse attention module performs the following operations:

[0022] First, the multi-scale fusion feature map is divided into blocks to obtain a feature vector sequence X=x1,x2,...,x N , Where N and n represent the length of the feature sequence, x i represents the i-th eigenvector in the sequence, represents the real number field, d is the feature dimension, Represents x i is a d-dimensional real vector, each element of which is a real number; Indicates that X is a real matrix of dimension n×d; each eigenvector is linearly transformed through three linear transformations Q=XW Q , K=XW K and V=XW V Get the query vector Q = q1, q2, ..., q N , key vector K=k1,k2,...,kN Sum vector V = v1, v2, ..., v N ,in W Q ,W K ,W V are learnable parameter matrices that are optimized through backpropagation during training to enable the model to adaptively learn representations of input sequences. W Q ,W K ,W V They are all d×d matrices formed by d-dimensional real vectors;

[0023] Next, a lightweight connection prediction module is used to estimate the connection score between each pair of feature vectors. The connection score reflects the semantic relevance of the two feature vectors. The connection prediction module performs the following operations:

[0024] Calculate the low-rank attention weight. The formula for low-rank attention weight is as follows:

[0025]

[0026] Among them, from the query Q and the downward projection key W down The outer product of K calculates the low-rank approximation of the attention matrix, W down is the learnable parameter matrix, n down Indicates the dimension reduction size, n indicates the length of the input feature sequence, W down K represents projecting the token dimension of K down to a lower dimension. Represents the feature dimension, softmax represents the normalization function, Represents the transpose operation of the matrix;

[0027] The low-rank attention weights are sparsely spaced by thresholding, and the formula is as follows:

[0028]

[0029] in, It represents the result of low-rank attention weight calculation between a pair of feature vectors i and j, τ represents the threshold. In low-rank attention sparsification, values ​​less than τ are directly discarded and zero values ​​are not stored.

[0030] By connecting the mask predictors, an upwardly projected sparse connection mask M is generated, which is expressed as:

[0031]

[0032] Among them, the connection mask predictor is the sparse projection matrix Wup Perform sparse matrix multiplication, i.e. W up It is a learnable parameter matrix that selects a limited number of similarity scores using the Top-k algorithm, that is, it selects the top k most relevant feature vectors as the objects of attention instead of calculating all possible pairings; it performs a binarization operation to obtain an upwardly projected sparse connection mask M, 1[·] represents binarization, which maps the elements in the subset to one and the other elements to zero. In the connection mask predictor, it is used to binarize the connection scores of each pair of tags, which represent their relevance to attention;

[0033] Then, under the guidance of the sparse connection mask M, the algorithm only calculates the non-zero elements of the full-rank attention weight A, that is, if each pair of feature vectors satisfies M between them ij =1, it means that they are similar enough to perform attention matching calculation. The formula for calculating the sparse full-rank attention matrix is ​​as follows:

[0034]

[0035] Finally, for each query vector i, the corresponding calculated output vector is: i,j∈[1,N], where when M ij ≠1, discard the corresponding Otherwise keep N is the length of the feature sequence, v j Represents a value vector V = v1, v2, ..., v N The representation of the jth element in , represents the weighted calculation result of attention between feature vectors i and j. The final calculation output of the sparse attention module of the entire instance-dependent is:

[0036] Furthermore, in step (3), the feature sequence obtained by the instance-dependent sparse attention module is The input data x is obtained after residual connection and normalization operation;

[0037] The gated linear control unit performs the following operations:

[0038] The input data x undergoes a linear transformation:

[0039] h=W·x+b1

[0040] Where h represents the intermediate vector, which is divided into two equal parts, a and b; W represents matrix multiplication, and b1 represents the addition of the bias term;

[0041] The input data x is multiplied by a Bernoulli distribution Bernoulli(φ(x)) through a GELU activation function, φ(x)=P(X≤x) to make it obey the standard normal distribution N(0,1), and a gate vector g is calculated:

[0042] g=σ(W g x+b g )

[0043] Among them, σ() represents the GELU activation function; W g is the weight of the gating mechanism, b g is the bias term of the gating mechanism;

[0044] By multiplying the gated vector g with b, we get the gated nonlinear part h gated =g⊙b, ⊙ represents the product;

[0045] Finally, the gated part is added to the linear part to obtain the output of the gated linear control unit GLU(x) = h gated +a.

[0046] Furthermore, in step (4), the feature sequence with position encoding can be regarded as a combination of a series of position embeddings and encoded image features. The multi-head self-attention module performs the following operations:

[0047] For each position, the Query, Key, and Value are calculated through three different linear transformations. These transformations perform a fully connected layer. The Query represents the features of the current position, and the Key and Value represent the features of other positions. For the Query of each position and the Key of other positions, the attention score is calculated. The attention score reflects the similarity between the current position and other positions. Finally, the attention weight is calculated and weighted summed. The attention score is scaled and then the attention weight is obtained through the softmax function. These weights determine the degree of attention each position pays to other positions. The attention weight is used to perform a weighted sum on the Value to obtain the self-attention output result.

[0048] Furthermore, in step (4), the query matrix Q in the instance-dependent sparse attention module is obtained by linearly transforming the encoder output feature sequence, and the key matrix K and value matrix V are obtained by linearly transforming the multi-head self-attention module with residual connection and normalization operation.

[0049] Furthermore, in step (6), the relative coordinate information of the predicted target bounding box is decoded and mapped back to the original image size; then a loss function is defined, including target category loss and target box coordinate position loss; the two losses are combined by weighted summation, and the predicted target box information is matched with the real information through the Hungarian algorithm.

[0050] Furthermore, the target category loss measures the difference between the predicted target category and the true target category. For each predicted target category, the category loss is calculated as follows:

[0051]

[0052] Among them, c i is the predicted target box category probability, N pos is the number of positive samples, pos is the index set of positive samples, 1 {i∈pos} is the indicator function, is the one-hot encoding of the corresponding true category label, p i is the predicted class probability;

[0053] The target box coordinate position loss measures the difference between the predicted target box coordinate position and the true target box coordinate position. For each predicted target box, the coordinate position loss is calculated as follows:

[0054]

[0055] Among them, b i is the coordinate position information of the predicted target box, N pos is the number of positive samples, pos is the index set of positive samples, 1 {i∈pos} is the indicator function, It is a smooth L1 loss function that balances the effects of large and small deviations. It is the coordinate position information of the corresponding real target frame.

[0056] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0057] (1) The deformable attention calculation requires offset prediction for each sampling point, which increases the computational complexity and memory consumption; the instance-dependent sparse attention used in the present invention does not require offset prediction, but directly uses instance features to calculate the attention weight, reducing computational overhead and improving computational efficiency.

[0058] (2) Deformable attention calculation may cause the attention position to deviate from the valid area, thereby reducing performance; the present invention uses a connection prediction mask module to limit the attention position, so that it can adaptively select the appropriate attention position, improve the quality of feature representation, avoid invalid areas, and thus have better detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flowchart of a sparse attention target detection method based on an improved DETR model provided in an embodiment of the present application;

[0060] Figure 2 This is a network structure diagram of the model in the embodiment of the present application;

[0061] Figure 3 This is a network structure diagram of the instance-dependent sparse attention module in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The present invention will be further described below with reference to the accompanying drawings.

[0063] like Figure 1 As shown, the embodiment of the present application provides a sparse attention target detection method based on the DETR improved model, including the following steps:

[0064] (1) Input the training data set into the backbone network Swin Transformer V1 and extract the three-layer feature maps C3, C4 and C5;

[0065] Combine Figure 2 , the original input feature map size is H×W×3, H represents the height of the image, W represents the width of the image; the backbone network Swin Transformer V1 extracts three layers of feature maps C3, C4 and C5, the sizes are and

[0066] The three-layer feature maps C3, C4 and C5 serve as the input of the subsequent multi-scale feature fusion module.

[0067] (2) Through the multi-scale feature fusion module, the three-layer feature maps C3, C4 and C5 are transformed into four feature layers, the four feature layers are fused and the corresponding relative position encoding information is added to obtain a multi-scale fused feature map;

[0068] Combine Figure 2 First, the three-layer feature maps C3, C4 and C5 are transformed into the size of and The feature map of the last layer C5 is transformed into a size of The feature map of s4 is obtained as the fourth feature layer.

[0069] Then, the coordinate information is added to the four feature layers. In order to distinguish the position information of feature points in different layers, relative position coordinates are introduced in multi-scale feature fusion. The position embedding method is to convert the absolute coordinates of the feature points of each layer into relative coordinates, that is, the offset of the point relative to the center of the input image.

[0070] Relative position encoding includes learnable scale-level embedding and position embedding;

[0071] The relative coordinates and scale information of the feature points in each layer are spliced ​​together to obtain a multi-scale fusion feature map.

[0072] (3) The multi-scale fusion feature map is used as the input of the encoder. The encoder is composed of six encoder layers. Each layer is mainly composed of an instance-dependent sparse attention module, a gated linear control unit, and a residual connection and normalization operation (Add&Norm) between them. The input feature fusion sequence is processed in turn by the instance-dependent sparse attention module, the residual connection and normalization operation and the gated linear control unit, and finally passes through a residual connection and normalization operation to obtain the output of a layer of encoder; repeat the six encoder traversals to obtain the encoder output feature map;

[0073] The encoder output feature map (output feature vector) is then used as the query input of the decoder.

[0074] Among them, combined Figure 2 and Figure 3 In the encoder part, the instance-dependent sparse attention module mainly consists of two parts: the connection prediction mask module M and the sparse attention module. The connection prediction mask module M is used to predict the connection probability between each pair of instances and save the connection score of each pair of instances; the sparse attention module sparsifies the attention weights according to the connection score of each pair of instances, retaining only a part of the important connections.

[0075] The connection prediction mask module M performs the following operations:

[0076] First, the multi-scale fusion feature map obtained in step (2) is used as the encoder input feature map, and the input feature map is divided into blocks to obtain a feature vector sequence X = x1, x2, ..., x N , Where N and n represent the length of the feature sequence, xi represents the i-th eigenvector in the sequence, represents the real number field, d is the feature dimension, Represents x i is a d-dimensional real vector, each element of which is a real number; Indicates that X is a real matrix of dimension n×d; each eigenvector is linearly transformed through three linear transformations Q=XW Q , K=XW K and V=XW V Get the query vector Q = q1, q2, ..., q N , key vector K=k1,k2,...,k N Sum vector V = v1, v2, ..., v N ,in W Q ,W K ,W V are learnable parameter matrices that are optimized through backpropagation during training to enable the model to adaptively learn representations of input sequences. W Q ,W K ,W V They are all d×d| matrices formed by d-dimensional real vectors;

[0077] Next, a lightweight connection prediction module is used to estimate the connection score between each pair of feature vectors. The connection score reflects the semantic relevance of the two feature vectors. The connection prediction module performs the following operations:

[0078] Calculate the low-rank attention weight. The formula for low-rank attention weight is as follows:

[0079]

[0080] Among them, from the query Q and the downward projection key W down The outer product of K calculates the low-rank approximation of the attention matrix, W down is the learnable parameter matrix, n down Indicates the dimension reduction size, which is set to 32; n indicates the length of the feature sequence; W down K represents projecting the token dimension of K down to a lower dimension. Represents the feature dimension, softmax represents the normalization function, Represents the transpose operation of the matrix;

[0081] The low-rank attention weights are sparsely spaced by thresholding, and the formula is as follows:

[0082]

[0083] in, represents the result of low-rank attention weight calculation between a pair of feature vectors i and j, τ represents the threshold, which is set to 0.05; in low-rank attention sparsification, values ​​less than 0.05 are directly discarded and zero values ​​are not stored;

[0084] The second step is to generate an upward projected sparse connection mask M through the connection mask predictor, which is expressed as:

[0085]

[0086] Among them, the connection mask predictor is the sparse projection matrix W up Perform sparse matrix multiplication, i.e. W up is a learnable parameter matrix that reduces computational and memory requirements by selecting a limited number of similarity scores using the Top-k algorithm, that is, selecting the top k most relevant feature vectors as the objects of attention rather than calculating all possible pairings. A binarization operation is performed to obtain an upwardly projected sparse connection mask M, where 1[·] represents binarization, which maps elements in a subset of feature vectors to one and all other elements to zero. In the connection mask predictor, it is used to binarize the connection scores of each pair of labels, which represent their relevance to attention. This processing allows subsequent calculations to focus only on those connections with the highest relevance to attention, further reducing computational and memory usage. In summary, the connection prediction mask module M stores the connection scores between each pair of feature vectors, with the connection scores for specific dependencies between them being one and the others being zero.

[0087] The sparse attention module performs the following operations:

[0088] Under the guidance of the sparse connection mask M, the algorithm only calculates the non-zero elements of the full-rank attention weight A, that is, if there is a pair of feature vectors, they satisfy M ij =1, it means that they have a dependency relationship and can perform attention matching calculation. The formula for calculating the sparse full-rank attention matrix is ​​as follows:

[0089]

[0090] Finally, for each query vector i, the corresponding calculated output vector is: i,j∈[1,N], where when M ij ≠1, discard the corresponding Otherwise keep N is the length of the feature sequence, vj Represents a value vector V = v1, v2, ..., v N The representation of the jth element in , represents the weighted calculation result of attention between feature vectors i and j. The final calculation output of the sparse attention module of the entire instance-dependent is:

[0091] Feature sequence obtained by instance-dependent sparse attention module After the residual connection and normalization operation, the input data x is obtained, and the input data x is input into the gated linear control unit;

[0092] The gated linear control unit performs the following operations:

[0093] The input data x undergoes a linear transformation (matrix multiplication W and bias addition b1):

[0094] h=W·x+b1

[0095] Where h represents the middle vector, which is divided into two equal parts, a and b;

[0096] The gating mechanism then multiplies the input data x by a Bernoulli distribution Bernoulli(φ(x)) through a GELU activation function, φ(x)=P(X≤x) to make it obey the standard normal distribution N(0,1), and calculates a gating vector g, which controls the information in the linear part b for filtering and control. The calculation process is as follows:

[0097] g=σ(W g x+b g )

[0098] Among them, σ() represents the GELU activation function; W g is the weight of the gating mechanism, b g is the bias term of the gating mechanism;

[0099] Secondly, by multiplying the gated vector g with b, we get the gated nonlinear part h gated =g⊙b, ⊙ represents the product;

[0100] Finally, the gated part is added to the linear part to obtain the output of the gated linear control unit GLU(x) = h gated +a, and then a layer of encoder output is obtained after residual connection and normalization operation.

[0101] (4) The encoder output feature map is used as the input of the decoder. The decoder is composed of six decoder layers stacked together. Each layer is mainly composed of Mutil-Head self-Attention, instance-dependent sparse attention module, gated linear control unit, and residual connection and normalization operations between them. The feature sequence with position encoding is input into Mutil-Head self-Attention. The output of Mutil-Head self-Attention is processed by residual connection and normalization operation, and then processed by instance-dependent sparse attention module, residual connection and normalization operation and gated linear control unit in sequence. Finally, it is processed by a residual connection and normalization operation to obtain the output of a layer of decoder. Repeat the decoder traversal six times to obtain the decoder output feature vector.

[0102] Combine Figure 2 In the decoder part, first, the feature sequence with position encoding is fed into Mutil-Head self-Attention. This feature sequence can be regarded as a combination of a series of position embeddings and encoded image features.

[0103] Mutil-Head self-Attention performs the following operations:

[0104] For each position, the query, key, and value are calculated through three different linear transformations. These transformations are fully connected. The query represents the characteristics of the current position, and the key and value represent the characteristics of other positions. For each position's query and the key of other positions, an attention score is calculated. The attention score reflects the similarity between the current position and other positions. Finally, the attention weight is calculated and weighted summed. The attention score is scaled and then passed through the softmax function to obtain the attention weight. These weights determine the degree of attention each position pays to other positions. The value is weighted summed using the attention weights to obtain the self-attention output. This aggregates information from other positions to generate a new representation for the current position.

[0105] The self-attention calculation results are passed through a residual connection and normalization (Add & Norm) operation as the input to the instance-dependent sparse attention module in the decoder. The input to the instance-dependent sparse attention module includes the sparse attention key and value, while the query input is the previous encoder output feature vector. After the instance-dependent sparse attention calculation, the subsequent steps of the decoder are the same as those of the encoder.

[0106] The instance-dependent sparse attention in step (3) and step (4) has the following differences:

[0107] The query matrix Q in the instance-dependent sparse attention module in step (4) is obtained by linearly transforming the encoder output feature sequence, while the key matrix K and value matrix V are obtained by linearly transforming the Mutil-Head self-Attention module with residual connection and normalization operation.

[0108] The subsequent operations are the same as step (3). The output of one layer of decoder is obtained through the gated linear control unit, residual connection and normalization (Add&Norm) operation. A total of six passes are performed to obtain the decoder output feature vector.

[0109] (5) The decoder output feature vector obtained in step (4) is passed through a linear layer and a multi-layer perceptron to predict the category and bounding box, respectively, to obtain a set of predicted targets, each of which contains category and bounding box coordinate information;

[0110] (6) Calculate the overall network loss between the predicted target set and the real target set, and optimize the model through back propagation;

[0111] Decode the predicted information. The predicted target bounding box coordinates appear in the form of relative coordinates and need to be decoded and demapped to the original image size. Then define a loss function to guide the model learning process. The loss function here consists of two parts: target category loss and target box coordinate position loss. The category loss and target box coordinate position loss are combined by weighted summation, and the predicted target box information is matched with the actual target box information using the Hungarian algorithm.

[0112] The target category loss measures the difference between the predicted target category and the true target category. For each predicted target category, the category loss is calculated as follows:

[0113]

[0114] Among them, c i is the predicted target box category probability, N pos is the number of positive samples, pos is the index set of positive samples, 1 {i∈pos} is the indicator function, is the one-hot encoding of the corresponding true category label, p i is the predicted class probability.

[0115] The target box coordinate position loss measures the difference between the predicted target box coordinate position and the true target box coordinate position. For each predicted target box, the coordinate position loss is calculated as follows:

[0116]

[0117] Among them, b i is the coordinate position information of the predicted target box, N pos is the number of positive samples, pos is the index set of positive samples, 1 {i∈pos} is the indicator function, smooth L1 It is a smooth L1 loss function that balances the effects of large and small deviations. It is the coordinate position information of the corresponding real target frame.

[0118] (7) Repeat steps (1) to (6) multiple times to obtain a trained target detection model, and use the trained target detection model for target detection.

[0119] The improved model in this paper utilizes an instance-dependent sparse attention mechanism to fine-tune attention to a portion of each object of interest while ignoring background and irrelevant areas. This sparsity improves computational efficiency while also making the model more robust to challenges such as occlusion and deformation. By introducing an instance-dependent mechanism, the model supports personalized processing of each object instance, selectively focusing on sampling points based on the characteristic information of each instance, thereby improving the accuracy of object detection.

[0120] This invention improves upon the Deformable DETR framework, better adapting to target detection tasks across various scenarios and scales, and boasts stronger generalization capabilities, making it a versatile target detection model. The target detection method provided by this invention has broad application scenarios and holds promise in areas such as object recognition, video surveillance, and autonomous driving.

Claims

1. A sparse attention target detection method based on the improved DETR model, characterized in that: include: (1) Input the training data set into the backbone network Swin Transformer V1 and extract the three-layer feature maps C3, C4 and C5; (2) Through the multi-scale feature fusion module, the three-layer feature maps C3, C4 and C5 are transformed into four feature layers, the four feature layers are fused and the relative position encoding information is added to obtain a multi-scale fused feature map; (3) The multi-scale fusion feature map is used as the input of the encoder. The encoder is composed of multiple encoder layers. Each layer is mainly composed of an instance-dependent sparse attention module, a gated linear control unit, and a residual connection and normalization operation between them. The input feature sequence is processed in turn by the instance-dependent sparse attention module, the residual connection and normalization operation, and the gated linear control unit. Finally, it undergoes a residual connection and normalization operation to obtain the output of a layer of encoder. Repeat the encoder multiple times to obtain the encoder output feature map; (4) The encoder output feature map is used as the input of the decoder. The decoder is composed of multiple decoder layers stacked together. Each layer is mainly composed of a multi-head self-attention module, an instance-dependent sparse attention module, a gated linear control unit, and residual connections and normalization operations between them. The feature sequence with position encoding is input into the multi-head self-attention module. The output of the multi-head self-attention module is processed by residual connection and normalization operations, and then processed by instance-dependent sparse attention module, residual connection and normalization operations and gated linear control unit in sequence. Finally, it undergoes a residual connection and normalization operation to obtain the output of a layer of decoder. Repeat the decoder multiple times to obtain the decoder output feature vector; (5) The decoder output feature vector is passed through a linear layer and a multi-layer perceptron to predict the category and bounding box, respectively, to obtain a set of predicted targets, each of which contains category and bounding box coordinate information; (6) Calculate the overall network loss between the predicted target set and the real target set, and optimize the model through back propagation; (7) Repeat steps (1) to (6) multiple times to obtain a trained target detection model; Step (2) includes: The three-layer feature maps C3, C4 and C5 are transformed into the size of and The feature map of the last layer, C5, is transformed into a size of The feature map of , as the fourth feature layer; The four feature layers are added with coordinate information. To distinguish the position information of feature points in different layers, relative position coordinates are introduced. The position embedding method is to convert the absolute coordinates of the feature points of each layer into relative coordinates. The relative coordinates and scale information of the feature points of each layer are spliced ​​together to obtain a multi-scale fusion feature map. In step (3), the instance-dependent sparse attention module performs the following operations: First, the multi-scale fusion feature map is divided into blocks to obtain a feature vector sequence X=x1,x2,...,x N , Where N and n represent the length of the feature sequence, x i represents the i-th eigenvector in the sequence, represents the real number field, d is the feature dimension, Represents x i is a d-dimensional real vector, each element of which is a real number; Indicates that X is a real matrix of dimension n×d; each eigenvector is linearly transformed through three linear transformations Q=XW Q , K=XW K and V=XW V Get the query vector Q = q1, q2, ..., q N , key vector K=k1,k2,...,k N Sum vector V = v1, v2, ..., v N ,in W Q ,W K ,W V are learnable parameter matrices that are optimized through backpropagation during training to enable the model to adaptively learn representations of input sequences. W Q ,W K ,W V They are all d×d matrices formed by d-dimensional real vectors; Next, a lightweight connection prediction module is used to estimate the connection score between each pair of feature vectors. The connection score reflects the semantic relevance of the two feature vectors. The connection prediction module performs the following operations: Calculate the low-rank attention weight. The formula for low-rank attention weight is as follows: Among them, from the query Q and the downward projection key W down The outer product of K calculates the low-rank approximation of the attention matrix, W down is the learnable parameter matrix, n down Indicates the dimension reduction size, n indicates the length of the input feature sequence, W down K represents projecting the token dimension of K down to a lower dimension. Represents the feature dimension, softmax represents the normalization function, Represents the transpose operation of the matrix; The low-rank attention weights are sparsely spaced by thresholding, and the formula is as follows: in, It represents the result of low-rank attention weight calculation between a pair of feature vectors i and j, τ represents the threshold. In low-rank attention sparsification, values ​​less than τ are directly discarded and zero values ​​are not stored. By connecting the mask predictors, an upwardly projected sparse connection mask M is generated, which is expressed as: Among them, the connection mask predictor is the sparse projection matrix W up Perform sparse matrix multiplication, i.e. W up It is a learnable parameter matrix that selects a limited number of similarity scores using the Top-k algorithm, that is, it selects the top k most relevant feature vectors as the objects of attention instead of calculating all possible pairings; it performs a binarization operation to obtain an upwardly projected sparse connection mask M, 1[·] represents binarization, which maps the elements in the subset to one and the other elements to zero. In the connection mask predictor, it is used to binarize the connection scores of each pair of tags, which represent their relevance to attention; Then, under the guidance of the sparse connection mask M, the algorithm only calculates the non-zero elements of the full-rank attention weight A, that is, if each pair of feature vectors satisfies M between them ij =1, it means that they are similar enough to perform attention matching calculation. The formula for calculating the sparse full-rank attention matrix is ​​as follows: Finally, for each query vector i, the corresponding calculated output vector is: Among them, when M ij ≠1, discard the corresponding Otherwise keep N is the length of the feature sequence, v j Represents a value vector V = v1, v2, ..., v N The representation of the jth element in , represents the weighted calculation result of attention between feature vectors i and j. The final calculation output of the sparse attention module of the entire instance-dependent is:

2. The sparse attention target detection method based on the improved DETR model according to claim 1 is characterized in that Step (1) includes: The original input feature map size is H×W×3, where H represents the height of the image and W represents the width of the image; The backbone network Swin Transformer V1 is used to extract three layers of feature maps C3, C4, and C5, with sizes as follows: and 3. The sparse attention target detection method based on the improved DETR model according to claim 1 is characterized in that Relative position encoding includes learnable scale-level embedding and position embedding.

4. The sparse attention target detection method based on the improved DETR model according to claim 1 is characterized in that In step (3), the feature sequence obtained by the instance-dependent sparse attention module The input data x is obtained after residual connection and normalization operation; The gated linear control unit performs the following operations: The input data x undergoes a linear transformation: h=W·x+b1 Where h represents the intermediate vector, which is divided into two equal parts, a and b; W represents matrix multiplication, and b1 represents the addition of the bias term; The input data x is multiplied by a Bernoulli distribution Bernoulli(φ(x)) through a GELU activation function, φ(x)=P(X≤x) to make it obey the standard normal distribution N(0,1), and a gate vector g is calculated: g=σ(W g ·x+b g ) Among them, σ() represents the GELU activation function; W g is the weight of the gating mechanism, b g is the bias term of the gating mechanism; By multiplying the gated vector g with b, we get the gated nonlinear part h gated =g⊙b, ⊙ represents the product; Finally, the gated part is added to the linear part to obtain the output of the gated linear control unit GLU(x) = h gated +a.

5. The sparse attention target detection method based on the improved DETR model according to claim 4 is characterized in that: In step (4), the feature sequence with position encoding can be regarded as a combination of a series of position embeddings and encoded image features. The multi-head self-attention module performs the following operations: For each position, the query, key, and value are calculated through three different linear transformations. These transformations are fully connected. The query represents the characteristics of the current position, and the key and value represent the characteristics of other positions. For each position's query and the key of other positions, an attention score is calculated. The attention score reflects the similarity between the current position and other positions. Finally, the attention weight is calculated and weighted summed. The attention score is scaled and then obtained through the softmax function to obtain the attention weight. These weights determine the degree of attention each position pays to other positions. Use the attention weight to perform weighted sum on the Value to obtain the self-attention output result.

6. The sparse attention target detection method based on the improved DETR model according to claim 5 is characterized in that: In step (4), the query matrix Q in the instance-dependent sparse attention module is obtained by linearly transforming the encoder output feature sequence, and the key matrix K and value matrix V are obtained by linearly transforming the multi-head self-attention module with residual connection and normalization operation.

7. The sparse attention target detection method based on the improved DETR model according to claim 6 is characterized in that: In step (6), the relative coordinate information of the predicted target bounding box is decoded and mapped back to the original image size; then the loss function is defined, including the target category loss and the target box coordinate position loss; the weighted sum of the two losses is combined, and the predicted target box information is matched with the real information through the Hungarian algorithm.

8. The sparse attention target detection method based on the improved DETR model according to claim 7 is characterized in that: The target category loss measures the difference between the predicted target category and the true target category. For each predicted target category, the category loss is calculated as follows: Among them, c i is the predicted target box category probability, N pos is the number of positive samples, pos is the index set of positive samples, 1 {i∈pos} is the indicator function, is the one-hot encoding of the corresponding true category label, p i is the predicted class probability; The target box coordinate position loss measures the difference between the predicted target box coordinate position and the true target box coordinate position. For each predicted target box, the coordinate position loss is calculated as follows: Among them, b i is the coordinate position information of the predicted target box, N pos is the number of positive samples, pos is the index set of positive samples, 1 {i∈pos} is the indicator function, It is a smooth L1 loss function that balances the effects of large and small deviations. It is the coordinate position information of the corresponding real target frame.

Images