Insect image detection method based on progressive attention and space decoupling

Abstract

The invention relates to a pest image detection method based on progressive attention and space decoupling. Compared with the prior art, the defect that field pest image detection precision is not high is overcome. The method comprises the following steps: acquiring a field pest image; constructing a pest image detection model; training a pest image detection model; acquiring a to-be-detected pest image; and obtaining a pest image detection result. According to the method, the target frame codes serve as prior knowledge to participate in updating of the target query, so that network convergence is faster, and the query of the next layer can perceive related information of the query of the previous layer through progressive attention target query updating. Meanwhile, based on the cross attention network of task decoupling, targeted target query is constructed for optimization of different tasks, target query focused on different tasks is more beneficial to fine network optimization, and the pest detection method based on progressive attention and space decoupling is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of disease image recognition, and in particular to a pest image detection method based on progressive attention and spatial decoupling. Background Art

[0002] With the rapid development of computer vision-related technologies, the application of computer technology for intelligent image analysis is becoming more and more widespread. Deep learning technology provides strong support for accurate identification, making pest image detection more efficient and accurate.

[0003] In actual applications, it is found that although the existing detection algorithms perform well for pest detection, there is still the problem of inconsistent model optimization paths, which can easily lead to problems such as missed detection and false detection in pest detection results. This is because these commonly used DETR-based detection methods need to learn the potential target distribution from a large amount of data, and the optimization direction of each decoding layer is independent and not completely consistent. In addition, the commonly used DETR-like algorithms use the same query feature for classification and positioning tasks, but the areas that classification and positioning features focus on may be different, and it is difficult to use the same query feature for better task optimization.

[0004] Therefore, how to construct a pest image detection method with high detection accuracy has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to solve the defect of low accuracy of field pest image detection in the prior art, and to provide a pest image detection method based on progressive attention and spatial decoupling to solve the above problem.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A pest image detection method based on progressive attention and spatial decoupling comprises the following steps:

[0008] 11) Obtaining field pest images: obtaining field pest images and preprocessing them to form a pest image dataset;

[0009] 12) Construct pest image detection model: Set the pest image detection model to include feature extraction network, feature encoding network, decoder and task head network;

[0010] 13) Training of pest image detection model: Use pest image dataset to train pest image detection model;

[0011] 14) Obtaining images of pests to be detected: obtaining images of field pests to be detected and performing preprocessing;

[0012] 15) Obtaining pest image detection results: Inputting the pre-processed field pest image to be detected into the trained pest image detection model to obtain the classification and positioning results of the field pests.

[0013] The construction of the pest image detection model comprises the following steps:

[0014] 21) The pest image detection model is set to include a feature extraction network, a feature encoding network, a decoder, and a task head network. The decoder includes 6 decoding layers. The input of the next decoding layer is the output of the previous decoding layer. The decoding layer includes a progressive attention network and a task decoupling network.

[0015] 22) Set up the feature extraction network:

[0016] The feature extraction network is set to be built based on the ResNet50 network, which is composed of superimposed multi-layer convolutional neural network layers, pooling layers, and activation function layers. The feature extraction network is used to extract the feature map of pests in the entire image. The input of the feature extraction network is the field pest image, and the output of the feature extraction network is the overall feature map of the field pest image.

[0017] 23) Setting a feature encoding network: the feature encoding network obtains the overall encoding features of the pest image according to the overall feature map, the input of the feature encoding network is the overall feature map, and the output of the feature encoding network is multiple target query vectors; the feature encoding network first encodes the multi-layer features of the overall feature map, maps it into a 256-dimensional feature vector, and generates a non-learned position code for the feature vector, the position code is obtained by encoding the position of each feature on the entire feature map through sine cosine functions; then the feature vector and the position code are added and further fused in the feature encoding network to obtain several encoded query vectors; the query vector is passed through a fully connected layer to obtain the scores corresponding to all queries, and the query vectors with the highest scores of each query vector are selected as the initialization target query of the decoder, and as the input of the progressive attention network of the first decoding layer of the decoder: query Q, key K and value V;

[0018] 24) Set a progressive attention network: the input of the progressive attention network is the Z query vectors with the highest scores among all target query vectors, and the output of the progressive attention network is the target query vector after the query interaction of different decoding layers. The progressive attention network enhances self-attention by encoding the spatial position of the target query and interacting with the target query of the previous layer;

[0019] 25) Set up a task decoupling network to decouple the target query.

[0020] The input of the task decoupling network is the target query vector of enhanced self-attention and the output feature vector of the feature encoding network. The output of the task decoupling network is two target queries for classification and positioning, that is, two different sets of output query vectors.

[0021] 26) Set a task head network to map the target query set to pest types and coordinates. The input of the task head network is two different output query vector sets of the task decoupling network. For the target query vectors in different sets, they pass through an FFN feedforward network respectively to predict the category and position of the target box.

[0022] The training of the pest image detection model includes the following steps:

[0023] 31) Input the pest image dataset into the feature extraction network of the pest image detection model. The feature extraction network uses the Resnet50 network to extract the overall feature map of dense pest images. Multi-layer features C2, C3, C4, and C5 are used to represent the features of each scale in the overall feature map. The C2, C3, C4, and C5 layer features are respectively subjected to 1x1 convolution operations to make the number of feature channels at each scale the same;

[0024] 32) Input the whole feature map into the feature encoding network. The feature encoding network adopts the Transformer encoder structure. The feature encoding network contains 6 encoding layers. Each encoding layer consists of a multi-head self-attention module and an FFN feedforward network. Both modules are connected with residual connection and Layer Norm layer. The input of the subsequent encoding layer is the feature vector output by the previous encoding layer. The feature encoding network outputs Z query vectors with the highest scores.

[0025] 33) Input the Z query vectors into the first layer of the decoding layer in the decoder:

[0026] 331) The input of the progressive attention network of the first decoding layer includes query Q, key K and value V, where Q, K and V are initialized to Z query vectors;

[0027] Query the corresponding target bounding box B. Through the interaction of target queries at different layers, the optimization direction of each decoding layer is kept consistent. First, the target bounding box B is encoded and the encoding is used as the spatial prior of progressive attention to participate in the update of query Q. The bounding box b is represented as [x, y, w, h]. The encoding E(b i ,b j ) is expressed as;

[0028]

[0029] Where x and y are the center points of the bounding box, w and h are the width and height of the bounding box, and xi is the x coordinate of the center point of the i-th query bounding box, x j is the x coordinate of the center point of the j-th query bounding box, w i is the width of the i-th query bounding box, y i is the y coordinate of the center point of the i-th query bounding box, y j is the y coordinate of the center point of the j-th query bounding box, h i is the height of the i-th query bounding box;

[0030] Convert it to 256-dimensional encoding through sine-cosine encoding:

[0031] PE(E,2i)=sin(E / 10000 2i / 64 ) (2)

[0032] PE(E,2i+1)=cos(E / 10000 2i / 64 ) (3)

[0033] Among them, PE is the spatial position information encoding, i represents the dimension index of the position encoding, and E is the encoding between bounding boxes;

[0034] By linearly changing the encoding, we get M weight matrices, where M is the number of attention heads:

[0035] W(B,B)=Linear(PE(B,B)) (4)

[0036] The shape of W(B,B) is N×N×M, where N is the number of target bounding boxes;

[0037] The specific formula of progressive attention is expressed as:

[0038]

[0039] l represents the number of the current decoding layer. When l=1, Q l-1 =Q,d k is the dimension of the vector, used to reduce the dot product range;

[0040] The output query vector after progressive attention is added to the input query vector and passed through the Layer Norm layer. The feature vector is then input into the FFN feed-forward network, which consists of two fully connected layers and a ReLu activation function.

[0041] 332) The input of the task decoupling network is the output query and target box of the progressive attention network. First, the query is split into two parts, one of which is used as the input query Q of the classification branch. cls , and the other part is used as the input query Q of the positioning branch loc ;

[0042] Q loc After a cross attention, we get Q′ loc , Q of the cross attention is the target query, K and V are all query feature vectors output by the feature encoding network, thus establishing the relationship between the target query and the global features of the entire image;

[0043] According to Q cls The corresponding target box obtains the features of the corresponding area from the feature extraction network. After a RoIAlign operation, the feature is passed through a 3x3 convolution and MLP layer to obtain the regional level feature Q f , the feature sine-cosine function is encoded and added to the input query as the input of the classification branch, and then Q is obtained through a cross attention. c ' ls ;

[0044] The specific formula is:

[0045] Q f =MLP(Conv(RoIAlign(F R ))) (6)

[0046] Q′ cls =CrossAttention(Q f +Q cls ) (7)

[0047] Q′ loc =CrossAttention(Q loc ) (8)

[0048] Among them, F R is the feature of the corresponding target box area on the feature extraction network, Q f To obtain regional features after 3x3 convolution and MLP layers;

[0049] Output query Q′ of the classification branch cls and the output query Q′ of the location branch loc After the residual and LayerNorm layer operations and the feed-forward network FFN, the two parts of the query are merged into a query set as the output query vector Q of the decoding layer. out ;

[0050] 34) The second, third, fourth, fifth and sixth layers of the decoding layer in the decoder repeat the process of the progressive attention network and task decoupling network in steps 331) and 332), where the input V and K of the progressive attention network are the same, and the output query Q of the previous decoding layer is out , Q is the input query Q of the progressive attention network in the previous decoding layer l-1 ;

[0051] 35) The task head network passes the target query of the classification branch and the positioning branch of the task decoupling network in the last decoding layer of the decoder through a feedforward network FFN to obtain the final classification and positioning results.

[0052] Beneficial Effects

[0053] The pest image detection method based on progressive attention and spatial decoupling of the present invention further improves the detection accuracy of pest images compared with the prior art. By encoding the target frame as prior knowledge to participate in the update of the target query, the network converges faster, and the target query is updated with progressive attention, so that the query of the next layer can perceive the relevant information of the query of the previous layer. At the same time, based on the cross-attention network with task decoupling, targeted target queries are constructed for the optimization of different tasks. Focusing on target queries of different tasks is more conducive to fine network optimization, realizing a pest detection method based on progressive attention and spatial decoupling.

[0054] The present invention also has the following advantages:

[0055] 1. The end-to-end pest detection method based on the Transformer architecture does not require manual setting of anchor frames and removes the post-processing NMS operation, which is conducive to the optimization training of the network;

[0056] 2. The present invention integrates the spatial position of the target box as prior knowledge, avoiding the problem that the network needs to implicitly learn spatial information from a large amount of data, which is more conducive to the rapid convergence of the model;

[0057] 3. The target query update based on progressive attention in the present invention is more suitable for maintaining the optimization consistency of each decoding layer, avoiding the reduction in accuracy caused by inconsistent optimization directions of target queries in different layers, so that the subsequent decoding layer can perceive the information of the target query of the previous decoding layer;

[0058] 4. The task decoupling network constructed in the present invention fully targets different tasks, performs task separation on cross-attention, and constructs target queries with enhanced features for classification tasks, making it easier for the network to optimize the classification and positioning of pests, thereby improving the accuracy of pest detection.

[0059] A progressive attention network is used to perceive the associations between queries in different decoding layers, so that the optimization paths between different decoding layers remain consistent during network training. The classification and localization query features of the DETR decoding layer are decoupled based on separated cross-attention, avoiding the complete sharing paradigm in DETR that hinders the model from learning better task-specific features to further improve its performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a method sequence diagram of the present invention;

[0061] Figure 2 It is a structural schematic diagram of the pest detection model involved in the present invention;

[0062] Figure 3 A schematic diagram of a progressive attention network involved in the present invention;

[0063] Figure 4 The figure is a diagram showing the effect of pest detection achieved by the method of the present invention. DETAILED DESCRIPTION

[0064] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0065] like Figure 1 As shown, the pest image detection method based on progressive attention and spatial decoupling described in the present invention comprises the following steps:

[0066] The first step is to obtain field pest images: obtain field pest images and preprocess them to form a pest image dataset.

[0067] The second step is to build a pest image detection model: Figure 2 As shown, the pest image detection model is set to include a feature extraction network, a feature encoding network, a decoder, and a task head network.

[0068] By optimizing the decoding process of the target query, the optimization direction of each decoding layer in the decoder is kept consistent, reducing the difficulty of query optimization between different decoding layers. At the same time, the cross-attention part is decoupled, so that the classification and positioning target queries are more focused on specific tasks, thereby improving the overall detection performance of the network.

[0069] (1) The pest image detection model is set to include a feature extraction network, a feature encoding network, a decoder, and a task head network. The decoder includes 6 decoding layers. The input of the subsequent decoding layer is the output of the previous decoding layer. The decoding layer includes a progressive attention network and a task decoupling network.

[0070] (2) Setting the feature extraction network:

[0071] The feature extraction network is set to be built based on the ResNet50 network, which is composed of superimposed multi-layer convolutional neural network layers, pooling layers and activation function layers. The feature extraction network is used to extract the feature map of pests in the entire image. The input of the feature extraction network is the field pest image, and the output of the feature extraction network is the overall feature map of the field pest image.

[0072] (3) Setting the feature encoding network: The feature encoding network obtains the overall encoding features of the pest image based on the overall feature map. The input of the feature encoding network is the overall feature map, and the output of the feature encoding network is multiple target query vectors. The feature encoding network first encodes the multi-layer features of the overall feature map, maps it into a 256-dimensional feature vector, and generates a non-learned positional encoding for the feature vector. The positional encoding is obtained by encoding the position of each feature on the entire feature map through sine-cosine functions. The feature vector and the positional encoding are then added and further fused in the feature encoding network to obtain a number of encoded query vectors.

[0073] The query vector is passed through a fully connected layer to obtain the scores corresponding to all queries. The Z query vectors with the highest scores for each query vector are selected as the initialization target queries of the decoder, which are used as the input of the progressive attention network of the first decoding layer of the decoder: query Q, key K and value V.

[0074] (4) The progressive attention network uses all feature points on the target query vector output by the feature encoding network as densely distributed initial queries, and selects 900 query vectors Q as the initialization vectors of the target query. The corresponding bounding box is used as the initialization of the target position box for updating. Figure 3 As shown, a progressive attention network is set: the input of the progressive attention network is the Z query vectors with the highest scores among all target query vectors, and the output of the progressive attention network is the target query vector after the query interaction of different decoding layers. The progressive attention network enhances self-attention by encoding the spatial position of the target query and interacting with the target query of the previous layer.

[0075] (5) Setting up a task decoupling network to decouple the target query.

[0076] The input of the task decoupling network is the target query vector of enhanced self-attention and the output feature vector of the feature encoding network. The output of the task decoupling network is two target queries for classification and positioning, that is, two different sets of output query vectors.

[0077] (6) A task head network is set to map the target query set into pest types and coordinates. The input of the task head network is two different output query vector sets of the task decoupling network. For the target query vectors in different sets, they are respectively passed through an FFN feed-forward network to predict the category and position of the target box.

[0078] The third step is to train the pest image detection model: use the pest image dataset to train the pest image detection model.

[0079] The self-attention and cross-attention parts of the decoder are reconstructed, and the target prior position is integrated for feature encoding to participate in the self-attention calculation of the query, thereby improving the convergence speed of the model. A progressive attention network is designed for query optimization and update, and the convolutional features of the target query corresponding area are fused with the query features to decouple the cross-attention, which is conducive to the optimization convergence of the model.

[0080] (1) The pest image dataset is input into the feature extraction network of the pest image detection model. The feature extraction network uses the Resnet50 network to extract the overall feature map of dense pest images. Multi-layer features C2, C3, C4, and C5 are used to represent the features of each scale in the overall feature map. The C2, C3, C4, and C5 layer features are respectively subjected to 1x1 convolution operations to ensure that the number of feature channels at each scale is the same.

[0081] (2) The overall feature map is input into the feature encoding network. The feature encoding network adopts the Transformer encoder structure. The feature encoding network contains 6 encoding layers. Each encoding layer consists of a multi-head self-attention module and an FFN feedforward network. Both modules are connected with residual connections and Layer Norm layers. The input of the subsequent encoding layer is the feature vector output by the previous encoding layer. The feature encoding network outputs the Z query vectors with the highest scores.

[0082] (3) Input the Z query vectors into the first layer of the decoding layer in the decoder:

[0083] A1) The input of the first progressive attention network of the decoding layer includes query Q, key K and value V, where Q, K and V are initialized to Z query vectors;

[0084] Query the corresponding target bounding box B. Through the interaction of target queries at different layers, the optimization direction of each decoding layer is kept consistent. First, the target bounding box B is encoded and the encoding is used as the spatial prior of progressive attention to participate in the update of query Q. The bounding box b is represented as [x, y, w, h]. The encoding E(b i ,b j ) is expressed as;

[0085]

[0086] Where x and y are the center points of the bounding box, w and h are the width and height of the bounding box, and x i is the x coordinate of the center point of the i-th query bounding box, x j is the x coordinate of the center point of the j-th query bounding box, w i is the width of the i-th query bounding box, y i is the y coordinate of the center point of the i-th query bounding box, y jis the y coordinate of the center point of the j-th query bounding box, h i is the height of the i-th query bounding box;

[0087] Convert it to 256-dimensional encoding through sine-cosine encoding:

[0088] PE(E,2i)=sin(E / 10000 2i / 64 ) (2)

[0089] PE(E,2i+1)=cos(E / 10000 2i / 64 ) (3)

[0090] Among them, PE is the spatial position information encoding, i represents the dimension index of the position encoding, and E is the encoding between bounding boxes;

[0091] By linearly changing the encoding, we get M weight matrices, where M is the number of attention heads:

[0092] W(B,B)=Linear(PE(B,B)) (4)

[0093] The shape of W(B,B) is N×N×M, where N is the number of target bounding boxes;

[0094] The specific formula of progressive attention is expressed as:

[0095]

[0096] l represents the number of the current decoding layer. When l=1, Q l-1 =Q,d k is the dimension of the vector, used to reduce the dot product range;

[0097] The output query vector after progressive attention is added to the input query vector and passed through the Layer Norm layer. The feature vector is then input into the FFN feed-forward network, which consists of two fully connected layers and a ReLu activation function.

[0098] A2) The input of the task decoupling network is the output query and target box of the progressive attention network. First, the query is split into two parts, one of which is used as the input query Q of the classification branch. cls , and the other part is used as the input query Q of the positioning branch loc ;

[0099] Q loc After a cross attention, we get Q′ loc , Q of the cross attention is the target query, K and V are all query feature vectors output by the feature encoding network, thus establishing the relationship between the target query and the global features of the entire image;

[0100] According to Q cls The corresponding target box obtains the features of the corresponding area from the feature extraction network. After a RoIAlign operation, the feature is passed through a 3x3 convolution and MLP layer to obtain the regional level feature Q f , the feature sine-cosine function is encoded and added to the input query as the input of the classification branch, and then Q is obtained through a cross attention. c ' ls ;

[0101] The specific formula is:

[0102] Q f =MLP(Conv(RoIAlign(F R ))) (6)

[0103] Q c ' ls =CrossAttention(Q f +Q cls ) (7)

[0104] Q′ loc =CrossAttention(Q loc ) (8)

[0105] Among them, F R is the feature of the corresponding target box area on the feature extraction network, Q f To obtain regional features after 3x3 convolution and MLP layers;

[0106] Output query Q of the classification branch c ' ls and the output query Q of the positioning branch l ' oc After the residual and LayerNorm layer operations and the feed-forward network FFN, the two parts of the query are merged into a query set as the output query vector Q of the decoding layer. out .

[0107] (4) The second, third, fourth, fifth and sixth layers of the decoding layer in the decoder repeat the process of the progressive attention network and task decoupling network in steps A1) and A2), where the input V and K of the progressive attention network are the same, and the output query Q of the previous decoding layer is out , Q is the input query Q of the progressive attention network in the previous decoding layer l-1 .

[0108] (5) The task head network passes the target query of the classification branch and the positioning branch of the task decoupling network in the last decoding layer of the decoder through a feedforward network FFN to obtain the final classification and positioning results.

[0109] The fourth step is to obtain the image of the pest to be detected: obtain the image of the field pest to be detected and perform preprocessing.

[0110] The fifth step is to obtain the pest image detection results: the pre-processed field pest image to be detected is input into the trained pest image detection model to obtain the classification and positioning results of the field pests.

[0111] The comparative experimental results of the present invention are shown in Table 1:

[0112] Table 1 Comparison of detection accuracy between the present invention and the existing method

[0113]

[0114] Table 1 shows a comparison of the detection results of the present invention and other prior art methods on the pest data set. Using the industry's advanced detection accuracy evaluation methods mAP, AP50, APS, APM, and APL, it can be seen from Table 1 that the detection accuracy of the present method is ahead of the prior art methods.

[0115] like Figure 4 As shown, it shows the visualization effect of the detection results of the present method, which realizes the effective detection of pests.

[0116] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A pest image detection method based on progressive attention and spatial decoupling, characterized in that: The following steps are involved: 11) Obtaining field pest images: obtaining field pest images and preprocessing them to form a pest image dataset; 12) Construct pest image detection model: Set the pest image detection model to include feature extraction network, feature encoding network, decoder and task head network; 13) Training of pest image detection model: Use pest image dataset to train pest image detection model; 14) Obtaining images of pests to be detected: obtaining images of field pests to be detected and performing preprocessing; 15) Obtaining pest image detection results: Inputting the pre-processed field pest image to be detected into the trained pest image detection model to obtain the classification and positioning results of the field pests.

2. The pest image detection method based on progressive attention and spatial decoupling according to claim 1, characterized in that: The construction of the pest image detection model comprises the following steps: 21) The pest image detection model is set to include a feature extraction network, a feature encoding network, a decoder, and a task head network. The decoder includes 6 decoding layers. The input of the next decoding layer is the output of the previous decoding layer. The decoding layer includes a progressive attention network and a task decoupling network. 22) Set up the feature extraction network: The feature extraction network is set to be built based on the ResNet50 network, which is composed of superimposed multi-layer convolutional neural network layers, pooling layers, and activation function layers. The feature extraction network is used to extract the feature map of pests in the entire image. The input of the feature extraction network is the field pest image, and the output of the feature extraction network is the overall feature map of the field pest image. 23) Setting a feature encoding network: the feature encoding network obtains the overall encoding features of the pest image according to the overall feature map, the input of the feature encoding network is the overall feature map, and the output of the feature encoding network is multiple target query vectors; the feature encoding network first encodes the multi-layer features of the overall feature map, maps it into a 256-dimensional feature vector, and generates a non-learned position code for the feature vector, the position code is obtained by encoding the position of each feature on the entire feature map through sine cosine functions; then the feature vector and the position code are added and further fused in the feature encoding network to obtain several encoded query vectors; the query vector is passed through a fully connected layer to obtain the scores corresponding to all queries, and the query vectors with the highest scores of each query vector are selected as the initialization target query of the decoder, and as the input of the progressive attention network of the first decoding layer of the decoder: query Q, key K and value V; 24) Set a progressive attention network: the input of the progressive attention network is the Z query vectors with the highest scores among all target query vectors, and the output of the progressive attention network is the target query vector after the query interaction of different decoding layers. The progressive attention network enhances self-attention by encoding the spatial position of the target query and interacting with the target query of the previous layer; 25) Set up a task decoupling network to decouple the target query. The input of the task decoupling network is the target query vector of enhanced self-attention and the output feature vector of the feature encoding network. The output of the task decoupling network is two target queries for classification and positioning, that is, two different sets of output query vectors. 26) Set a task head network to map the target query set to pest types and coordinates. The input of the task head network is two different output query vector sets of the task decoupling network. For the target query vectors in different sets, they pass through an FFN feedforward network respectively to predict the category and position of the target box.

3. The pest image detection method based on progressive attention and spatial decoupling according to claim 1, characterized in that: The training of the pest image detection model includes the following steps: 31) Input the pest image dataset into the feature extraction network of the pest image detection model. The feature extraction network uses the Resnet50 network to extract the overall feature map of dense pest images. Multi-layer features C2, C3, C4, and C5 are used to represent the features of each scale in the overall feature map. The C2, C3, C4, and C5 layer features are respectively subjected to 1x1 convolution operations to make the number of feature channels at each scale the same; 32) Input the whole feature map into the feature encoding network. The feature encoding network adopts the Transformer encoder structure. The feature encoding network contains 6 encoding layers. Each encoding layer consists of a multi-head self-attention module and an FFN feedforward network. Both modules are connected with residual connection and Layer Norm layer. The input of the subsequent encoding layer is the feature vector output by the previous encoding layer. The feature encoding network outputs Z query vectors with the highest scores. 33) Input the Z query vectors into the first layer of the decoding layer in the decoder: 331) The input of the progressive attention network of the first decoding layer includes query Q, key K and value V, where Q, K and V are initialized to Z query vectors; Query the corresponding target bounding box B. Through the interaction of target queries at different layers, the optimization direction of each decoding layer is kept consistent. First, the target bounding box B is encoded and the encoding is used as the spatial prior of progressive attention to participate in the update of query Q. The bounding box b is represented as [x, y, w, h]. The encoding E(b i ,b j ) is expressed as; Where x and y are the center points of the bounding box, w and h are the width and height of the bounding box, and x i is the x coordinate of the center point of the i-th query bounding box, x j is the x coordinate of the center point of the j-th query bounding box, w i is the width of the i-th query bounding box, y i is the y coordinate of the center point of the i-th query bounding box, y j is the y coordinate of the center point of the j-th query bounding box, h i is the height of the i-th query bounding box; Convert it to 256-dimensional encoding through sine-cosine encoding: PE(E,2i)=sin(E / 10000 2i / 64 ) (2) PE(E,2i+1)=cos(E / 10000 2i / 64 ) (3) Among them, PE is the spatial position information encoding, i represents the dimension index of the position encoding, and E is the encoding between bounding boxes; By linearly changing the encoding, we get M weight matrices, where M is the number of attention heads: W(B,B)=Linear(PE(B,B)) (4) The shape of W(B,B) is N×N×M, where N is the number of target bounding boxes; The specific formula of progressive attention is expressed as: l represents the number of the current decoding layer. When l=1, Q l-1 =Q,d k is the dimension of the vector, used to reduce the dot product range; The output query vector after progressive attention is added to the input query vector and passed through the Layer Norm layer. The feature vector is then input into the FFN feed-forward network, which consists of two fully connected layers and a ReLu activation function. 332) The input of the task decoupling network is the output query and target box of the progressive attention network. First, the query is split into two parts, one of which is used as the input query Q of the classification branch. cls , and the other part is used as the input query Q of the positioning branch loc ; Q loc After a cross attention, we get Q l ' oc , Q of the cross attention is the target query, K and V are all query feature vectors output by the feature encoding network, thus establishing the relationship between the target query and the global features of the entire image; According to Q cls The corresponding target box obtains the features of the corresponding area from the feature extraction network. After a RoIAlign operation, the feature is passed through a 3x3 convolution and MLP layer to obtain the regional level feature Q f , the feature sine-cosine function is encoded and added to the input query as the input of the classification branch, and then Q is obtained through a cross attention. c ' ls ; The specific formula is: Q f =MLP(Conv(RoIAlign(F R ))) (6) Q′ cls =CrossAttention(Q f +Q cls ) (7) Q′ loc =CrossAttention(Q loc ) (8) Among them, F R is the feature of the corresponding target box area on the feature extraction network, Q f To obtain regional features after 3x3 convolution and MLP layers; Output query Q′ of the classification branch cls and the output query Q′ of the location branch loc After the residual and Layer Norm layer operations and the feed-forward network FFN, the two parts of the query are merged into a query set as the output query vector Q of the decoding layer. out ; 34) The second, third, fourth, fifth and sixth layers of the decoding layer in the decoder repeat the process of the progressive attention network and task decoupling network in steps 331) and 332), where the input V and K of the progressive attention network are the same, and the output query Q of the previous decoding layer is out , Q is the input query Q of the progressive attention network in the previous decoding layer l-1 ; 35) The task head network passes the target query of the classification branch and the positioning branch of the task decoupling network in the last decoding layer of the decoder through a feedforward network FFN to obtain the final classification and positioning results.

Images