A lightweight pest recognition method based on Transformer structure

Abstract

The present invention relates to a lightweight pest recognition method based on a Transformer structure, which belongs to the field of deep learning and comprises the following steps: S1: extracting shallow features of pest images using a focused fast downsampling module; S2: extracting global feature information in a deep feature map using a multi-head self-attention module; S3: adding local feature sensitivity and scale invariance information to the deep feature map using local convolution; S4: performing feature splicing on the global feature information with the local feature sensitivity and scale invariance information to obtain a pest image rich in semantic information, sending the global feature information to a multi-layer perceptron, and performing feature fitting on the fused feature tensor; S5: reducing the gradient vanishing problem of the network through residual connection, and integrating the information in the channel through point-by-point convolution; and S6: using a pooling mechanism to classify the finally calculated feature representation through a classification module.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and relates to a lightweight pest recognition method based on a Transformer structure. Background Art

[0002] With the growing greenhouse effect in recent years, agricultural and forestry pests and diseases have become increasingly serious, leading to increasing food losses. Therefore, refined prevention and control of pests and diseases is a key measure to address the current decline in grain production. Currently, manual pest identification is widely used. To reduce production costs and alleviate agricultural workload, research is needed on efficient and low-cost automatic pest identification algorithms.

[0003] Traditional automatic pest identification algorithms are primarily based on machine learning techniques from computer vision and consist of three stages: pest image preprocessing, feature extraction, and feature classification. These methods typically begin by enhancing salient areas and removing background from the image. Features such as the subject's color, texture, and shape are then extracted. Finally, image classification is performed using methods such as support vector machines (SVMs), Adaboost, and artificial neural networks (ANNs). These methods suffer from low accuracy and robustness, and their excessive reliance on manual techniques during feature extraction limits their applicability.

[0004] With the rapid development of deep learning in computer vision, the pest automatic identification system based on convolutional neural networks has achieved end-to-end automatic feature extraction with high speed and accuracy, and has become the current mainstream pest automatic identification solution. In order to achieve accurate identification of pests in complex background images, current research mainly focuses on the following three aspects: (1) Pest identification method based on saliency features. This method uses methods such as graph-based visual saliency (GBVS), patch-saliency map algorithm (Patch-Saliency Map Algorithm) or cluster-based saliency detection to increase the threshold of the salient area in the input image to achieve the purpose of highlighting the salient area, and then uses convolutional neural networks to extract and classify its features. Since pest images often contain complex background interference such as color and texture, conventional saliency algorithms are difficult to extract high-level semantic information from them. (2) Pest identification method combined with attention mechanism. This method adds channel or spatial attention mechanism to the convolutional neural network to enhance the network's feature extraction ability to achieve better classification results. However, while the attention mechanism increases the number of model parameters, it does not adequately handle the spatial relationship between features. At the same time, pest datasets generally have the characteristics of large intra-class gaps, small inter-class gaps, and complex backgrounds. Convolutional neural networks with small receptive fields find it difficult to extract high-level semantic information from them, and cannot achieve a good balance between accuracy and speed. (3) Pest recognition method that integrates multiple models. This method first trains different models, and then obtains a new pest recognition model by integrating the structure and weights to achieve a pest recognition model with higher accuracy. However, this method only uses accuracy as the only indicator, resulting in a large number of model parameters, high training costs, and often requires the use of some high-performance computing equipment to be deployed in the cloud. It has certain limitations in places with weak signals such as large-scale farmland and mountainous areas. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to use the Transformer structure that can perform global feature modeling on the input to add a global receptive field to the model and enhance the model's ability to resist complex backgrounds. However, due to its own lack of local sensitivity and scale invariance, the simple transformer structure requires a huge amount of parameters and training costs, making it difficult to achieve lightweight. Therefore, this paper designs the shallow and deep layers of model feature extraction from the two aspects of parameter quantity and global receptive field, and proposes a pest recognition network with a lightweight Transformer structure (Transformer based Pest Classification Network, TPCNet). TPCNet designs the downsampling structure in the shallow layer from the aspects of parameter quantity, model complexity, floating point operations (FLOPs) and memory access cost (MAC), thereby reducing the inference time of the shallow layer. Transformer is used in the deep layer to increase the network receptive field, and convolution is used to make up for the shortcomings of the simple Transformer structure that lacks locality and scale invariance.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A lightweight pest recognition method based on a Transformer structure includes the following steps:

[0008] S1: Extract shallow features of pest images using a focused fast downsampling module;

[0009] S2: Use a multi-head self-attention module to extract global feature information from deep feature maps;

[0010] S3: Use local convolution to add local feature sensitivity and scale invariance information to deep feature maps;

[0011] S4: The global feature information is combined with the local feature sensitivity and scale invariance information to obtain a pest image with rich semantic information. The image is then fed into a multi-layer perceptron to perform feature fitting on the fused feature tensor.

[0012] S5: Reduce the vanishing gradient problem of the network through residual connections and use point-by-point convolution to integrate information in the channel;

[0013] S6: The final calculated feature representation is pooled and classified through the classification module.

[0014] Furthermore, in step S1, the specific operations of the focused fast downsampling module are as follows:

[0015] S11: Separate the downsampling operation from the local feature extraction operation;

[0016] S12: Replace the convolution operation with a stride of 2 with a slicing operation that takes alternate pixel values ​​and then concatenates the channels.

[0017] S13: Use 1*1 point-by-point convolution to reduce and integrate the number of feature map channels and information after the downsampling operation;

[0018] S14: Use 3*3 convolution to increase the ability to extract local features.

[0019] Furthermore, in step S2, the multi-head self-attention mechanism of the Transformer structure is used to calculate the mutual correlation between feature pixels, and the expression is:

[0020] X a =MultiHead(X ai )=Concat(head1,...,head i )W o (1)

[0021]

[0022] Q=X ai W Q ,K=X ai W K ,V=X ai W V (3)

[0023] Where W Q ,W K ,W V ,W o Represents the query, key, value and weighted sum projection weight matrix in the self-attention mechanism, X ai Represents the input feature tensor, X a Represents the output feature tensor.

[0024] Furthermore, step S4 specifically includes: cross-joining the global feature information with the local feature sensitivity and scale invariance feature information to increase the fusion capability of different features. The formula is:

[0025] X c =Conv2d(X ci ) (4)

[0026] X m =Concat(X c ,X a )+X i (5)

[0027] X o =Concat(MLP(LN(X m ))+(LN(X m ))) (6)

[0028] where X i represents the residual connection to prevent the gradient from disappearing, MLP(·) represents the multi-layer perceptron, and LN(·) represents the normalization function. c Represents the local feature sensitivity and scale invariance feature tensor obtained after convolution, X m Represents the tensor that fuses global feature information and local feature information, X o Represents the tensor obtained after nonlinear mapping of the multilayer perceptron.

[0029] Furthermore, in step S5, the initial input feature tensor X is subjected to a residual branch spanning global feature extraction and local feature supplementation, and the gradient vanishing problem is solved by feature splicing. Subsequently, a channel-wise point-by-point convolution is used to fuse the features. The formula is:

[0030] X = Conv2d(Concat(X o ,X)) (7).

[0031] Furthermore, in step S6, the feature vector is pooled into two dimensions before final recognition, and then the most likely category is obtained through the Linear layer. The formula is:

[0032] X o =avgpool(X) (8)

[0033] Pre=Linear(X o ) (9)

[0034] After the Linear layer, a two-dimensional tensor Pre is obtained, and the final prediction output is obtained through this two-dimensional vector.

[0035] The beneficial effects of the present invention are as follows: the present invention redesigns the feature extraction modules of the shallow and deep layers respectively. The focused fast downsampling module designed by the present invention is used in the shallow layer to accelerate downsampling while increasing the model's ability to extract local features. The inverted residual Transformer module designed by the present invention is used in the deep layer to model global features while using local convolution branches to increase the module's sensitivity to local features and scale invariance, thereby achieving lightweight modules and enhancing the generalization ability of the model.

[0036] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0038] Figure 1 This is the overall flow chart of the lightweight pest identification method based on the Transformer structure of the present invention;

[0039] Figure 2 This is a diagram of the lightweight pest recognition network framework based on the Transformer structure of the present invention;

[0040] Figure 3 This is a schematic diagram of the framework of the focused fast downsampling module of the present invention;

[0041] Figure 4 This is a schematic diagram of the inverted residual Transformer module framework of the present invention;

[0042] Figure 5 Comparison of the parameters and recognition accuracy of the present invention with other lightweight models;

[0043] Figure 6 The test speed of the present invention and other lightweight models on different platforms. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0046] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0047] Figure 1 This is the overall flow chart of a lightweight pest identification method based on Transformer structure involved in the present invention. Figure 1 Further description. The present invention mainly includes the steps of:

[0048] Step 1: Use the focused fast downsampling module designed by the present invention to extract shallow features of the pest image.

[0049] Step 2: Use the Multi-Head Self Attention Module to extract global feature information from the deep feature map.

[0050] Step 3: Use local convolution to add local feature sensitivity and scale invariance information to the deep feature map.

[0051] Step 4: The global feature information is concatenated with the local feature sensitivity and scale invariance information to obtain a pest image with rich semantic information. The image is then fed into a multi-layer perceptron (MLP) for feature fitting of the fused feature tensor.

[0052] Step 5: Reduce the vanishing gradient problem of the network through residual connections. Then use point-by-point convolution to integrate the information in the channel.

[0053] Step 6: The final calculated feature representation is passed through the classification module to obtain the classification result.

[0054] Figure 2 This is a lightweight pest recognition network framework diagram based on the Transformer structure in the present invention. Figure 2 The structural principle of the present invention is further explained. As shown in the figure, the model of the present invention has 9 layers, and each layer is mainly composed of two modules:

[0055] (1) Focus on the fast downsampling module FFDM

[0056] In order to make the model lightweight, the present invention uses the focused fast downsampling module constructed by the present invention in the first three layers, which quickly reduces the size of the feature map while increasing the shallow local feature extraction capability of the network. At the same time, the design of the focused fast downsampling module reduces the number of parameters and the amount of calculation, making the overall network more lightweight.

[0057] like Figure 3 The figure shows the focused fast downsampling module constructed by the present invention. H×W×C , which samples and concatenates the interval pixels of the input tensor, and finally obtains a 2x downsampled output tensor X∈R with 4 times the number of channels (H / 2)×(W / 2)×(C / 2) This structure only performs arithmetic operations on the input tensor and therefore does not contain any parameters. Furthermore, this structure can map continuous planar features to channels, reducing feature loss during downsampling. To enhance information interaction between channels while controlling the number of output channels, the present invention uses point-by-point convolution to reduce the tensor's channels while enabling cross-channel feature interaction and information integration. Finally, grouped convolution is used to extract local features from the tensor, increasing the model's sensitivity to local features.

[0058] (2) Inverted Residual Transformer Module IRTM

[0059] To address the difficulty of convolutional neural networks with small receptive fields in extracting high-level semantic information from pest images with complex backgrounds, this paper leverages the multi-head self-attention architecture within the Transformer architecture to model global features of pest images. However, a simple Transformer architecture struggles to fit the dataset, so this paper adds a convolutional branch to enhance local sensitivity and scale invariance.

[0060] like Figure 4 The figure shows the inverted residual Transformer module constructed by the present invention. For the input tensor X∈R H×W×C , a 3×3 standard convolution is used to encode the local spatial information of the input tensor; then a 1×1 point-by-point convolution is used to map the tensor to a high-dimensional space by learning the linear combination of the input channels to obtain X I ∈R H×W×d , providing more feature information for subsequent modules. In order to use locality to reasonably compensate for global dependencies within the inverted residual Transformer module, we modify the internal structure of the Transformer as follows Figure 4The modification shown in the figure enables the Transformer structure to model local and global dependencies separately by adding a parallel convolution branch to the multi-head self-attention structure. I , after being normalized by LayerNorm, it is output to the channel splitting module to split the channel.

[0061] x a =split(LN(X I [0:d / r]))

[0062] x c =split(LN(X I [d / r:d]))

[0063] where x a ∈R H×W×(d / r) and x c =R H×W×(d-d / r) They represent the input tensors used by the multi-head self-attention structure branch to establish long-distance dependencies on features and the input tensors used by the convolution branch to model local features; split represents the splitting function of the input tensor channel; LN represents normalization; and r represents the splitting rate. In order to use the multi-head self-attention structure to globally model pest features, this paper uses the conversion function Img2seq with a splitting rate of p to transform the three-dimensional tensor x a Convert to a 2D tensor After that, it is sent to the multi-head self-attention structure to calculate:

[0064]

[0065]

[0066] Where W Q ,W K ,W V ,W o Represents the query, key, value, and weighted sum projection weight matrix in the self-attention mechanism. Two-dimensional output tensor Represents the feature correlation matrix of the input tensor. a Restored to a three-dimensional tensor by the Seq2Img function Get the three-dimensional global feature representation of pest features; and the local feature representation tensor of the convolution branch After splicing, we get a tensor X containing local feature representation and global feature representation M ∈R H×W×d :

[0067] X M =Concat(X C,X A )+X I

[0068] where X I Represents the residual connection to prevent gradient disappearance; it is worth noting that the present invention swaps the order of the processed local representation and global representation tensors so that all tensors processed by the model can alternately model local and global features. M After LN and MLP functions, the input tensor X is concatenated to obtain:

[0069] X' o =Concat((MLP(LN(X M ))+X M ),X)

[0070] In order to reduce the final output tensor dimension of the IRTM module, the present invention uses a point-by-point convolution to transform the tensor X' o ∈R H ×W×(c+d) Mapped to a low-dimensional space, the channel features are aggregated, and the final output is X o ∈R H×W×C .

[0071] Depend on Figure 5 and Figure 6 It can be seen from the simulation results that the model designed by the present invention is lightweight and has a high recognition accuracy.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A lightweight pest identification method based on a Transformer structure, characterized by: The following steps are involved: The pest image is input into the focused fast downsampling module, and sampling and splicing operations are performed on the interval pixels, specifically including: separating the downsampling operation from the local feature extraction operation; The convolution operation with a stride of 2 is replaced by a slicing operation that takes interval pixel values ​​and then splices channels. 1*1 point-by-point convolution is used to reduce and integrate the number of feature map channels and information after the downsampling operation. 3*3 convolution is used to increase the ability to extract local features. The tensor processed by the focused fast downsampling module X Input inverted residual Transformer module, using a 3 The standard convolution of 3 encodes the local spatial information of the input tensor; then uses 1 1 point-by-point convolution, which maps the tensor to a high-dimensional space by learning a linear combination of input channels ; Add a parallel convolution branch to the multi-head self-attention structure to convert the tensor After LayerNorm normalization, the output is sent to the channel splitting module to split the channel, and the result is: in Represents the input tensor of the multi-head self-attention structure branch used to establish long-range dependencies on features; Represents the input tensor of the convolution branch for modeling local features; split represents the splitting function of the input tensor channel; LN represents normalization; represents the split ratio; Use the conversion function Img2seq with a split rate of p to convert the three-dimensional tensor Convert to a 2D tensor After that, it is sent to the multi-head self-attention structure to calculate: in Represents the projection weight matrix of the weighted sum in the self-attention mechanism, a two-dimensional output tensor Represents the feature correlation matrix of the input tensor; The two-dimensional tensor Restored to a three-dimensional tensor by the Seq2Img function Obtaining a three-dimensional global feature representation of pest characteristics; Will Input the convolution branch, add local feature sensitivity and scale invariance information, and obtain the local feature representation tensor ; Will and Perform splicing to obtain a tensor containing local feature representation and global feature representation : in It is a residual connection used to prevent the gradient from disappearing; The tensor After normalization and multi-layer perceptron and input tensor X After splicing, we get: in represents a multilayer perceptron, represents the normalization function; Use a channel-wise convolution to transform the tensor Mapped to a low-dimensional space, the channel features are aggregated at the same time, and the final output is ; The final calculated feature representation is used in a pooling mechanism and classified through a classification module.

2. The lightweight pest identification method based on the Transformer structure according to claim 1 is characterized by: Before the final recognition, the feature tensor is pooled into two dimensions and then passed through the Linear layer to obtain the most likely category. The formula is: After the Linear layer, a two-dimensional tensor Pre is obtained, through which the final prediction output is obtained.

Images