An end-to-end pest detection method with feature compensation and local information enhancement

Abstract

The application discloses an end-to-end pest detection method with feature compensation and local information enhancement, and belongs to the field of deep learning. First, a Transform network framework based on feature representation compensation and cross-shaped window attention mechanism is established to extract multi-scale features of an image. Second, regional grid self-attention is used to replace even-numbered multi-head self-attention modules in a Transform model encoder to obtain feature information of the image. Finally, the first K encoder features in the last layer are selected to initialize the position query of a Transform decoder with contrastive denoising, and the content query remains a learnable parameter. Compared with a convolution-based method, the technical scheme of the application can achieve a higher average precision mean in a pest detection task.

Description

Technical Field

[0001] This invention relates to deep learning technology, specifically to an end-to-end pest detection method with feature compensation and local information enhancement. Background Technology

[0002] The Transformer is a deep learning model based on a self-attention mechanism, initially used for machine translation in natural language processing tasks. Proposed in 2017 by Vaswani et al., it eliminates the order and positional constraints of traditional recurrent neural networks and convolutional neural networks by introducing a self-attention mechanism. Later, Dosovitskiy et al. proposed ViT (VisionTransformer), which divides the original image into several non-overlapping image patches of the same size and inputs them into a pure Transformer model, achieving significant results in the field of image processing.

[0003] A common approach to pest detection is the use of Convolutional Neural Networks (CNNs). Thanks to their ability to recognize complex patterns in visual data, CNN-based algorithms have become important tools in agricultural applications. However, CNNs still have some limitations that affect the accuracy of pest detection. Their reliance on downsampling can lead to the loss of valuable low-level information. For pests, many key identifying features, such as antennae, legs, or specific body markings, are often small and complex. These important details may be lost during feature extraction, making it difficult for the model to distinguish between different types of pests, thus affecting the accuracy and reliability of pest detection. Furthermore, the feature extraction capability of CNN-based models is a significant area of ​​concern. While CNNs excel at extracting local features, their ability to obtain global context from input data is limited, resulting in poor global feature representation. This limitation hinders the model's ability to understand long-range dependencies in image data, which is crucial for accurate pest detection.

[0004] The AF-RCNN model outperforms Fast R-CNN and YOLO in pest identification, thanks to its feature fusion and innovative matching strategy. However, its performance may degrade due to insufficient labeled samples for some pest categories, requiring data augmentation and mobile platform integration. The T-CNN model, by independently classifying crops and diseases, can improve accuracy in real-world environments and effectively identify crop diseases; however, its performance in real-world scenarios is poor, necessitating improvements in model robustness and adaptability. Summary of the Invention

[0005] This invention proposes an end-to-end pest detection method based on Transformer, featuring feature compensation and local information enhancement. The aim is to compensate for the loss of feature information during downsampling by adding a feature representation compensation module to the traditional cross-shaped window Transformer, thereby improving the performance of the pest detection model without increasing complexity. At the same time, some multi-head self-attention in the Transformer encoder is replaced with regional raster self-attention to capture global and local information of the image, further improving the performance of the pest detection model.

[0006] The technical solution adopted in this invention includes:

[0007] An end-to-end pest detection method with feature compensation and local information enhancement includes the following steps:

[0008] A Transformer network framework based on feature representation compensation and cross-shaped window attention mechanism is established to extract multi-scale features of the input image;

[0009] A Transformer encoder using regional raster self-attention flattens the multi-scale features and uses them as input to the encoder to extract global and local feature information.

[0010] Using a contrastive denoising Transformer decoder, the first K terms of the feature information are extracted to initialize the position query of the encoder, while the content query remains a learnable parameter.

[0011] Furthermore, the Transformer network framework includes: firstly, using a convolutional label embedding layer to transform the pest image into image patch labels, the purpose of which is to model the local context and transition from low-level edges to higher-level semantic information; then, inputting the image patch labels into the Transformer network framework of the feature representation compensation and cross-shaped window self-attention mechanism, the network contains four stages, each stage having Ni consecutive cross-shaped window Transformer blocks, and a feature representation compensation module is added in the second, third, and fourth stages, by combining the enhanced features obtained by the feature representation compensation module with the output of the last stage of the cross-shaped window Transformer to obtain multi-scale features of the input image.

[0012] Furthermore, the convolutional label embedding layer includes: C convolutional channels, n×n kernel size, where n is the height and width of the convolutional kernel, s stride, p padding, and H×W×3 dimensions for the input image data, where H is the height of the input image data and W is the width of the input image data. The size of the image block label is...

[0013] Furthermore, the Transformer network framework also includes: the network comprises four stages, where the output of two consecutive stages is F. i-1 and F i F represents the feature map output at a certain stage of the model, and the subscript i represents the index of that stage, which can be 1, 2, 3, or 4. First, F is reconstructed using upsampling and a 1×1 convolution. i Then calculate F i and F i-1 The feature differences are then normalized to between 0 and 1 using the Sigmoid function, and the result can be interpreted as F. i-1 Weight matrix of loss information right and F i-1 By performing matrix multiplication, we can obtain the enhanced F. i-1 , for enhanced F i-1 Perform average pooling and combine with F i Summing these values ​​yields the enhanced F. i .

[0014] Furthermore, the Transformer network framework also includes: sparse long-range residual connections added in the third stage, as shown in the following formula.

[0015]

[0016] Among them, F i It is the output of the i-th layer, x i It is the input of the i-th layer, z(x) i ) is for input x i The operations performed include: self-attention, normalization, and ReLU activation. i mod k means that the integer i is modulo k, where k represents the number of layers traversed and the sparsity.

[0017] Furthermore, the Transformer encoder with regional grid self-attention includes replacing the multi-head attention module in the even-numbered Transformer block with the regional grid self-attention module.

[0018] Furthermore, the region-based raster self-attention module in the Transformer encoder includes: uniformly dividing the feature map along the channel dimension, and performing row-level and column-level attention calculations within the raster window, such as when the input feature map X∈R H×W×C It is divided into two independent parts in terms of channel dimension. and Row-level self-attention is performed on feature map Xr, while column-level self-attention is performed on feature map Xc. Finally, the outputs of these two parallel groups are concatenated together, R.H×W×C Let X be a multidimensional real space with height H, width W, and number of channels C. Let X be the input feature map, H be the height of the input feature map, W be the width of the input feature map, C be the number of channels of the input feature map, Xr be the feature map used for row grid self-attention, and Xc be the feature map used for column grid self-attention.

[0019] First, a Transformer network framework (FRC-CSWin) based on Feature Representation Compensation and Cross-Shaped Window Self-Attention is used to extract multi-scale features from the image data. This model can calculate the feature information differences between each stage and compensate for the information loss of the feature map during downsampling, thus obtaining features containing more detailed information. Then, the feature information is flattened and input into the Transformer encoder. In this process, multi-head self-attention and region grid self-attention are organically combined in the encoder to extract features containing both local and global information. Finally, in the last layer, the features from the top K encoders are selected to initialize the position query of the Transformer decoder with contrastive de-noising, while the content query remains as learnable parameters.

[0020] Before using FRC-CSWin Transformer to extract multi-scale features from image data, it is necessary to use Convolutional Token Embedding (CNN) to transform the pest image into image patch tokens. The purpose is to model the local context and transition from low-level edges to high-level semantic information.

[0021] The feature representation compensation process can be interpreted as follows: Let the output of the CSWin Transformer in two consecutive stages be F. i-1 and F i First, F is reconstructed using upsampling and 1×1 convolution. i Then calculate F i and F i-1 The feature differences are then normalized to between 0 and 1 using the Sigmoid function, and the result can be interpreted as F. i-1 The weight matrix of the loss information is denoted as right and F i-1 By performing matrix multiplication, we can obtain the enhanced F. i-1 For the enhanced F i-1 Perform average pooling and combine with F iSumming these values ​​yields the enhanced F. i .

[0022] This invention adds sparse long-range residual connections to the third stage of the CSWin Transformer. These connections allow information to propagate between specified layers, rather than permeating every layer. This allows for more efficient management of parameters and computational requirements while still enjoying the advantages of the Transformer model. The formula is expressed as follows:

[0023]

[0024] When processing pest image data, capturing long-range dependencies is crucial. Traditional convolutional neural networks struggle to capture these dependencies due to limitations in their local receptive fields. While self-attention mechanisms effectively address this issue, they neglect fine-grained information from the input data. Therefore, this invention proposes a Regional Pale-shaped Self-Attention (RPSA). This method allows the model to capture not only global features of the image but also local information, making it more suitable for pest detection tasks. We construct the Transformer encoder by replacing the Multi-Head Self-Attention (MSA) module in even-numbered Transformer blocks with the Regional Pale-shaped Self-Attention module.

[0025] DN-DETR is highly effective in stabilizing training and accelerating convergence. With the help of DN queries, the model learns to make predictions based on nearby anchors with ground truth boxes. However, it lacks the ability to make "objectless" predictions for anchors without nearby objects. To address this issue, Zhang et al. proposed a contrastive denoising (CDN) method to reject useless anchors. In this section, we follow the work of Zhang et al. and introduce CDN into the Transformer decoder.

[0026] The beneficial effects of this invention are:

[0027] This invention proposes an end-to-end pest detection method based on Transformer, featuring feature compensation and local information enhancement. It compensates for the feature information loss caused by downsampling in traditional cross-shaped Transformer models and, without increasing model complexity, fuses global and local features to capture more complex information. Compared to traditional methods, this invention better understands semantic information in images, reducing false positives and false negatives. Due to the complexity and variability of pest detection scenarios, traditional methods are often sensitive to factors such as lighting changes and background interference. This invention, through feature compensation and local information enhancement strategies, effectively addresses these challenges, improving the robustness of the pest detection algorithm. Pest damage to crops is a significant issue in agricultural production. The pest detection method proposed in this invention is universal and can be applied to different types of crops and pests, providing farmers with an efficient and accurate pest monitoring tool, facilitating timely implementation of appropriate control measures, and improving crop yield and quality. Ultimately, our invention achieved a mean average precision (mAP) of 35.6% on the IP102 dataset, placing it at a leading level. Attached Figure Description

[0028] Figure 1 This is a schematic flowchart of the end-to-end pest detection method described in this invention;

[0029] Figure 2 This is a schematic diagram of the overall framework of the FRC-CSWin Transformer model described in this invention;

[0030] Figure 3 This is a schematic diagram of the feature representation compensation module of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of two consecutive Transformer blocks in the Transformer encoder described in this invention;

[0032] Figure 5 This is a schematic diagram of the region grid-shaped self-attention described in this invention; Detailed Implementation

[0033] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in specific implementation examples.

[0034] 1. This invention primarily uses the IP102 dataset for training and testing the model. The IP102 dataset is a large benchmark dataset used for pest classification, containing more than 75,000 images and 102 categories. Additionally, 19,000 images are labeled with bounding boxes for object detection. This invention divides the IP102 dataset in a 7:3 ratio for model training and testing.

[0035] 2. We first use Convolutional Token Embedding (CNN) to transform pest images into image patch tokens. The aim is to model the local context and transition from low-level edges to high-level semantic information. Formally, given an RGB image x∈R... H×W×3 As input to the aforementioned FRC-CSWin, we use the function f(·) to map x to a new image patch marker with C channels. This function is essentially a convolutional layer with an n×n kernel, a stride of s, and padding of p. The size of the new image patch marker is calculated as follows:

[0036]

[0037] H is the height of the original image data, and W is the width of the original image data. new W is the height marked for the new image patch. new The width of the new image patch is marked, in this example n=7, s=4, p=0.

[0038] 3. The overall architecture of FRC-CSWin Transformer is as follows: Figure 2 As shown, the specific steps are as follows:

[0039] Step 1) Input the pest image into the FRC-CSWin Transformer to obtain the feature maps F of two adjacent stages. i-1 and F i F i-1 For feature maps with larger sizes, F i This corresponds to a smaller feature map. F i-1 A richer representation that includes target-specific information, while F i This incorporates higher-level semantic information about the pest target. However, F i There is also some missing information, especially in parts related to small pests and fine-grained information about pests.

[0040] Step 2) Apply upsampling and 1×1 convolution operations to F i Reconstruct the data, then calculate the result with F. i-1The feature differences are normalized to between 0 and 1 using the Sigmoid function. The normalized feature differences can be understood as F... i-1 Weights for lost information. The formula is as follows:

[0041]

[0042] Conv 1×1 (·) represents a convolutional layer with a kernel size of 1×1, and UpSampling(·) represents an upsampling operation.

[0043] Step 3) Attach the weights to F i-1 This enhances the fine-grained information of the layer. Furthermore, the same information is added to F through average pooling. i Feature map. Enhanced F i-1 and F i It can be represented in the following forms

[0044]

[0045]

[0046] Avgpooling(·) represents the average pooling operation.

[0047] It should be added that the enhanced feature information obtained in stages 2, 3, and 4, combined with the feature information output by the Transformer in stage 4, constitutes the multi-scale feature information of the input data. The number of Transformer heads in the four stages of the CSWin Transformer are 2, 4, 8, and 16, respectively, and the number of Transformer blocks are 1, 2, 21, and 1, respectively.

[0048] 4. This invention adds sparse long-range residual connections to the third stage of the CSWin Transformer. These connections allow information to propagate between specified layers, rather than permeating every layer. This allows for more efficient management of parameters and computational requirements while still enjoying the advantages of the Transformer model. The formula is expressed as follows:

[0049]

[0050] Among them, F i It is the output of the i-th layer, x i It is the input of the i-th layer, z(x) i ) is for input x i The series of operations performed (self-attention, normalization, ReLU activation) are represented by k, which indicates the number of layers traversed and the sparsity, i.e., a residual connection is applied every k layers.

[0051] 5. We construct the Transformer encoder by replacing the multi-head self-attention (MSA) module in the even-numbered Transformer blocks with the regional grid self-attention module. As Figure 4 shown, two consecutive Transformer blocks respectively include multi-head self-attention and regional grid self-attention, and a 2-layer MLP with the GELU non-linear activation function is used between them. The LayerNorm (LN) layer is applied before each self-attention module and each MLP, and the residual connection is used after each module.

[0052] The grid self-attention window is as Figure 5 shown by the shadow in r Each grid region with side length l (l < H & l < W) consists of s c interleaved rows and s r l + s c l - s r s c interleaved columns, covering a region containing s r s c s H×W×C markers. l is the side length of the grid region, s M is the number of columns, s is of length s r l + s c l - s r s c , and the real number field with channel number C.

[0053] To further enhance the model's ability to obtain local information, we evenly divide the feature map along the channel dimension and perform row-level and column-level self-attention calculations within the grid window respectively. As Figure 5 shown in the middle part, the input feature map X ∈ R H×W×C is divided into two independent parts along the channel dimension and This means that the K heads in RPSA also need to be divided into two parallel groups (each group has K / 2 heads), where one group performs row-level self-attention and the other group performs column-level self-attention. Finally, the outputs of these two parallel groups are concatenated together, and the expression is as follows:

[0054] RPS - Attention(X) = Concat(head1,..., head k )W O (7)

[0055]

[0056] Among them W O ∈R C×C R is a commonly used projection matrix. C×C Let k be a real number field of height C and width C, onto the target output dimension, and let k be the index of the attention head in the multi-head self-attention process. k For the k-th head in multi-head self-attention, RPS-Attention represents the region grid self-attention operation, and R-Attention... k C-Attention represents the computation of row-level attention. k This indicates the calculation of column-level attention. For example... Figure 4 As shown, the forward propagation of two consecutive Transformer blocks in the encoder can be described as follows:

[0057]

[0058]

[0059]

[0060]

[0061] in and Y n Let represent the output features of the MSA and MLP for the nth block, respectively; MSA(·) and RPSA(·) represent the use of conventional multi-head self-attention and region-based grid self-attention, respectively; LN(·) represents layer normalization. The MLP consists of two fully connected layers used to perform additional nonlinear transformations on the output of the self-attention layer. In this example, the side length of the region-based grid self-attention is l = [20, 10, 5], s r =s c =7. In addition, the size of the data input to the multi-head self-attention block is 1×1045.

[0062] 6. In the last layer, select the top K encoder features to initialize the position query of the Transformer decoder with contrastive de-noising, while the content query remains as learnable parameters for predicting the bounding box and species of pests.

[0063] Finally, it should be noted that the above examples are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Therefore, although the present invention has been described in detail with reference to the above examples, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention. All technical solutions and improvements that do not depart from the spirit and scope of the invention should be covered within the scope of the claims of the present invention.

[0064] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0065] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An end-to-end pest detection method with feature compensation and local information enhancement, characterized in that, The method comprises the following steps: A Transformer network framework based on feature representation compensation and cross-shaped window attention mechanism is established to extract multi-scale features of an input image; A regional raster self-attention Transformer encoder is used to flatten the multi-scale features and take them as inputs of the encoder to extract global and local feature information of the input; A contrastive denoising Transformer decoder is used to take the first K items of the feature information to initialize the position query of the encoder, and the content query is still a learnable parameter; The Transformer network framework comprises the following steps: first, a convolution marker embedding layer is used to convert a pest image into image block markers, which aims to model local context and transition from low-level edges to higher-level semantic information; then, the image block markers are input into the Transformer network framework based on feature representation compensation and cross-shaped window self-attention mechanism, the network comprises four stages, each stage has Ni consecutive cross-shaped window Transformer blocks, and a feature representation compensation module is added in the second, third and fourth stages, the enhanced features obtained by the feature representation compensation module are combined with the output of the last stage of the cross-shaped window Transformer to obtain the multi-scale features of the input image.

2. The method of claim 1, wherein, The convolution mark embedding layer comprises a convolution channel number , a convolution kernel size , n is a height and width of the convolution kernel, a step is , padding is , a size of an input image is , H is a height of input image data, W is a width of input image data, and a size of the image block mark is .

3. The method of claim 1, wherein, The Transformer network framework also includes: the network consists of four stages, where the output of two consecutive stages is... and F represents the feature map output at a certain stage of the model, and the subscript i represents the index of that stage, which can be 1, 2, 3, or 4. First, upsampling and... Convolutional reconstruction Then calculate and The characteristic differences, and then use The function normalizes the feature differences to and The result obtained can be interpreted as follows: Weight matrix of loss information ;right and Performing matrix multiplication yields an enhanced result. , for enhancement Perform average pooling and Summation yields an enhanced result. .

4. The method of claim 3, wherein, The Transformer network framework further comprises the following step: a sparse long-range residual connection is added in the third stage, and the formula is as follows: ； wherein, is the output of the layer, is the input of the layer, is the operation performed on the input , the operation comprising: self-attention, normalization, ReLU activation, i mod k denotes the integer mod operation on , denotes the number of layers and sparsity spanned.

5. The method of claim 1, wherein, The regional raster self-attention Transformer encoder comprises the following step: the multi-head attention module in the even-numbered Transformer blocks is replaced by the regional raster self-attention module.

6. The method of claim 5, wherein, The region-wise self-attention module in the region-wise self-attention Transformer encoder includes: dividing the feature map along the channel dimension uniformly, and respectively performing attention calculation at the row level and the column level within the grid window, such as the input feature map is divided into two independent parts on the channel dimension and , the row-level self-attention is performed on the feature map Xr, and the column-level self-attention is performed on the feature map Xc, and finally the outputs of the two parallel groups are spliced together, R H×W×C is a multi-dimensional real number space with a height of , a width of , and a channel number of , X is an input feature map, H is the height of the input feature map, W is the width of the input feature map, C is the channel number of the input feature map, Xr is a feature map for performing row grid self-attention, and Xc is a feature map for performing column grid self-attention.

Images