Tomato pest and disease detection and identification method based on deep learning

Abstract

This invention discloses a deep learning-based method for detecting and identifying tomato pests and diseases. The method comprises: using multiple tomato pest and disease images and the labeled pest locations in each image as a first sample training set, performing data augmentation, and then training a Transformer-based YOLOX object detection network; labeling the pest categories to which the pest locations in the tomato pest and disease images belong; using this as a second sample training set, performing data augmentation, and then training a rotation-invariant Fisher discriminant CNN network; and using the YOLOX object detection network and the rotation-invariant Fisher discriminant CNN network to locate the pest locations in the tomato images and identify their categories. This invention maximizes the advantages of both networks to assist in tomato pest and disease management, achieving detection and location of pest targets and identification of pest categories.

Description

Technical Field

[0001] The present invention relates to the technical field of tomato disease and pest detection, and specifically to a tomato disease and pest detection and identification method based on deep learning. Background Art

[0002] As a common fruit and vegetable, tomatoes inevitably develop pests and diseases during cultivation, impacting quality and reducing yields. Building an intelligent network for detecting and identifying tomato pests and diseases, enabling timely identification and targeted prevention, is crucial for increasing tomato yields and also offers insights into pest and disease control for other fruits and vegetables.

[0003] Traditional manual pest control is time-consuming and labor-intensive, and can lead to inaccurate judgments and time lags due to external interference and human physiological limitations. With the development of computer vision technology, some traditional machine learning methods have been widely applied to pest and disease identification, significantly improving efficiency. Classic machine learning methods include support vector machines (SVMs), K-means clustering, and naive Bayes. SVMs can be used as binary classifiers. For tomato leaf disease detection, gray-level co-occurrence matrices are used for feature extraction, and then SVMs are used to further classify the extracted features into healthy and infected leaves. Histogram of oriented gradients features can also be combined with SVMs to detect and identify rice diseases. The K-means clustering algorithm can perform image segmentation and then be combined with neural network tools for classification. Seven features extracted from clusters—contrast, correlation, energy, homogeneity, mean, standard deviation, and variance—are used as input to the neural network for disease identification and classification. The Naive Bayesian approach can be used to implement techniques and methods for the automatic identification and diagnosis of crop diseases, constructing a module for the classification and recognition of corn leaf lesions. To address issues such as low image quality, noise and shadows, cluttered backgrounds, and varying image textures, the system first preprocesses and extracts various features. Then, a multilayer perceptron (MLP) is used to integrate four classifiers: Random Forest (RF), Support Vector Machine (SVM), Logistic Regression (LR), and K-Nearest Neighbor (K-NN). This classifies images of tomato leaf diseases under field and laboratory conditions with an accuracy of 95.98%. However, due to the high dimensionality and complex feature structure of images, the process of effectively extracting and representing features is relatively complex when using machine learning algorithms for image classification. Furthermore, machine learning algorithms struggle to process similar samples from different angles, distances, and lighting conditions.

[0004] In recent years, deep learning technology has been widely researched and applied in various fields. Convolutional neural networks (CNNs) are particularly effective in processing multidimensional array data and are particularly suitable for the detection and recognition of data such as images and videos. Thanks to their powerful learning capabilities and ability to automatically extract image features, deep learning technology is more accurate and efficient than traditional machine learning techniques in identifying plant pests and diseases. For example, improvements have been made to the traditional AlexNet model, using a convolutional neural network model that combines batch normalization with global pooling to identify various leaf diseases. Disease classifiers have been constructed using two well-known deep models, AlexNet and GoogleNet. Experimental results demonstrate that CNN models offer superior performance in tomato disease classification compared to shallow models that require manual feature processing. Another example is the lightweight multi-branch residual network (LMBRNet), which uses four branches with different convolutional kernel sizes to extract multiple disease features from tomato leaves across different dimensions and receptive fields, achieving both accuracy and speed in identifying tomato leaf diseases.

[0005] In addition to identifying the type of tomato pests and diseases, detecting the location of tomato pests and diseases early in the disease's development on tomato leaves, locating the precise location of lesions and pests, and implementing timely control measures can effectively prevent large-scale pest and disease outbreaks in tomato cultivation bases. Object detection algorithms can be divided into two categories based on their workflow: two-stage algorithms, such as Faster R-CNN, first generate candidate boxes as samples, then search for objects and adjust the bounding boxes; single-stage algorithms, such as SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once), directly transform the object boundary localization problem into a regression problem. The model only processes the image once to obtain bounding box coordinates and class probabilities. Two-stage disease detection algorithms include: Incorporating the Region Proposal Network (RPN) architecture into the Faster R-CNN algorithm for real-time detection of rice leaf diseases, which can very accurately locate the target location; and using a modified Mask R-CNN combined with an ensemble neural network to detect the type and infected area of ​​tomato leaf diseases. Single-stage algorithms, such as the YOLOv5 model, incorporate the SE (Squeeze-and-Excitation) module, employing an attention mechanism to extract key features and effectively detect areas of tomato virus disease. The differences between the two approaches also lead to different performances. The two-stage approach offers superior detection and localization accuracy, but suffers from slower detection speed. The single-stage approach sacrifices high accuracy but achieves significantly faster speeds. A corn pest and disease detection method, based on the single-stage object detection YOLOv3 algorithm, improves on the K-means algorithm, achieving a balance between average loss reduction and object detection accuracy. However, using deep learning models for image detection and recognition also has limitations, such as the need for large and diverse datasets for training, and the high risk of erroneous detection and recognition results due to abnormal object poses caused by varying image angles.

[0006] While traditional target detection methods have classification capabilities, they focus more on object search. Most research focuses on target location, while classification capabilities have not been further developed. Even some traditional CNN networks, which have demonstrated strong image classification performance, only have a single classification function. In tomato cultivation and management, target detection and classification must be applied simultaneously. To achieve optimal results in real-world applications, a novel deep learning-based tomato pest and disease detection and identification method is needed, taking into account the small, densely packed targets and complex natural backgrounds of tomato pests and diseases. This method can detect and locate pest and disease targets and identify their species. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a tomato disease and pest detection and identification method based on deep learning to address the deficiencies of the above-mentioned existing technologies. This tomato disease and pest detection and identification method based on deep learning constructs a Transformer-based YOLOX target detection network and a rotation-invariant Fisher discriminant CNN classification and recognition network; it maximizes the use of the respective advantages of the two networks to assist in the management of tomato diseases and pests, realizes the detection and positioning of pest targets, and identifies the categories of pests and diseases.

[0008] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0009] The tomato pest and disease detection and identification method based on deep learning includes the following steps:

[0010] (1) Collect multiple tomato pest and disease images and mark the location of the pests in each tomato pest and disease image. The original tomato pest and disease images and the tomato pest and disease images with marked pest and disease locations are used as the first sample training set;

[0011] (2) Labeling the pest and disease location in the tomato pest and disease image obtained in step (1) with the pest and disease category to which the pest and disease location belongs, and using the tomato pest and disease image with the pest and disease location labeled and the pest and disease category to which it belongs as the second sample training set; collecting multiple healthy tomato plant images, and using the multiple healthy tomato plant images and the health category to which they belong as the second sample training set;

[0012] (3) Perform data enhancement on the first sample training set and the second sample training set respectively to obtain a new first sample training set and a new second sample training set;

[0013] (4) Using the new first sample training set to train the Transformer-based YOLOX target detection network, a trained pest and disease target detection model is obtained;

[0014] (5) Using the new second sample training set to train the rotation-invariant Fisher discriminant CNN network, a trained pest and disease classification and recognition model is obtained;

[0015] (6) Inputting the tomato image to be detected into the pest and disease target detection model to locate the location of the pest and disease in the tomato image;

[0016] (7) The image in which the location of the pests and diseases in the tomato image is located is input into the pest and disease classification and recognition model, thereby identifying the category of the pests and diseases in the tomato image.

[0017] As a further improved technical solution of the present invention, the data enhancement is to perform four random angle rotation operations on the sample.

[0018] As a further improved technical solution of the present invention, the Transformer-based YOLOX target detection network includes a Swin Transformer backbone network, a BiFPN enhanced feature extraction network and an output network.

[0019] As a further improved technical solution of the present invention, the Swin Transformer backbone network includes stage one, stage two, stage three and stage four;

[0020] The first stage includes a block embedding module and two Swin Transformer modules for downsampling and dimensionality transformation of the input image through convolution; the second stage includes a block merging module and two Swin Transformer modules for performing shape transformation and block merging on the input image; the third stage includes a block merging module and six Swin Transformer modules for performing shape transformation and block merging on the input image; the fourth stage includes a block merging module and two Swin Transformer modules for performing shape transformation and block merging on the input image.

[0021] As a further improved technical solution of the present invention, the BiFPN enhanced feature extraction network includes an upsampling path and a downsampling path;

[0022] The second stage of the Swin Transformer backbone network is used as the upper feature layer, the third stage as the middle feature layer, and the fourth stage as the deep feature layer. The output of the deep feature layer is reshaped to obtain a deep feature map. The output of the middle feature layer is reshaped to obtain a middle feature map. The output of the upper feature layer is reshaped to obtain an upper feature map. The deep feature map, the middle feature map, and the upper feature map are all input to the BiFPN enhanced feature extraction network.

[0023] The upsampling path of the BiFPN enhanced feature extraction network is as follows: the deep feature map is fused with the middle feature map after a 2-fold upsampling operation, and then the features are extracted through a convolution operation to obtain the output of the first convolution operation. The output of the first convolution operation is then upsampled by 2 times and fused with the upper feature map. The fused result is then convolved to obtain the output of the first effective feature layer.

[0024] The downsampling path of the BiFPN enhanced feature extraction network is as follows: the output of the first effective feature layer is downsampled by 2 times, fused with the output of the first convolution operation and the middle-layer feature map, and then the fusion result is convolved to obtain the output of the second effective feature layer; the output of the second effective feature layer is downsampled by 2 times, fused with the deep feature map, and then the fusion result is convolved to obtain the output of the third effective feature layer.

[0025] As a further improved technical solution of the present invention, the output network includes three decoupling detection module layers, a splicing layer, and a transposition layer. The three decoupling detection module layers are simultaneously connected to the splicing layer, and the splicing layer is connected to the transposition layer. The decoupling detection module layer is composed of a decoupling detection head, a splicing layer, and a shape transformation layer connected in sequence.

[0026] The decoupled detection head consists of a convolutional layer, a BN layer, and a SiLU activation function layer.

[0027] As a further improved technical solution of the present invention, the rotation-invariant Fisher discriminant CNN network includes 5 convolutional layers, a fully connected layer, a rotation-invariant Fisher discriminant layer and a softmax classification layer;

[0028] The first convolutional layer includes two convolution operations, the second convolutional layer includes two convolution operations, the third convolutional layer includes three convolution operations, the fourth convolutional layer includes three convolution operations, the fifth convolutional layer includes three convolution operations, and the fully connected layer includes three convolution operations; there are maximum pooling layers between the first and second convolutional layers, between the second and third convolutional layers, between the third and fourth convolutional layers, between the fourth and fifth convolutional layers, and between the fifth convolutional layer and the fully connected layer.

[0029] As a further improved technical solution of the present invention, the objective function of the rotationally invariant Fisher discriminant CNN network is:

[0030]

[0031] Among them, W RI Denotes all weights of the rotation-invariant Fisher discriminant CNN network, B RI Represents all biases of the rotation-invariant Fisher discriminant CNN network, X RI ={x i |x i∈X∪TX}, represents the new second sample training set after data enhancement, where X represents the second sample training set before data enhancement, T = {T1, T2, T3, T4} represents 4 rotation operations, TX represents the sample training obtained after all samples in X have undergone 4 rotation operations, and the total number of the second sample training set before data enhancement is N, X RI The number is N*5; λ1, λ2 and λ3 represent trade-off parameters, and <a,b> represents the inner product of a and b. Represents the training sample x i The category label of O8(x i ) represents the training sample x i The output features of the softmax classification layer, O7(x i ) represents the training sample x i The output features of the rotation-invariant Fisher discriminant layer, Represents the training sample x i The average output feature of the sample obtained after 4 rotation operations is the rotation-invariant Fisher discriminant layer, n j is the number of samples in the jth category, m j Represents all samples in the jth category The average feature representation of , there are 7 categories in total, m is all samples X of all categories RI The average feature representation of , tr is the trace of the matrix, which represents the sum of the diagonal elements of the matrix.

[0032] The beneficial effects of the present invention are:

[0033] The present invention uses Swin Transformer as the backbone network to construct a Transformer-based YOLOX target detection network. The network independently calculates self-attention within a sliding window (Shifted Windows, SW), improves computational efficiency, comprehensively considers global feature information, and achieves more effective small-target pest and disease detection and positioning. In order to solve the problem of pest and disease image shape changes caused by different angles and the problem of high similarity between different pests and diseases and variable characteristics of similar pests and diseases, a rotation-invariant Fisher discriminant layer is added to the traditional CNN architecture. By adding two regularization constraints to the objective function of the model to train this newly added rotation-invariant Fisher discriminant layer, a rotation-invariant Fisher discriminant CNN classification and recognition network is constructed. The respective advantages of the two networks are maximized to assist in the management of tomato pests and diseases, and the detection and positioning of pest and disease targets and the identification of categories are achieved.

[0034] Tomato is one of the three major world traded vegetables and occupies an important position in the global vegetable trade. With the continuous increase in the demand for tomatoes, the total production and planting scale of tomatoes in the world are constantly expanding. Using computers or artificial intelligence technology to assist in the prevention and control of tomato diseases and pests has become a feasible strategy. Therefore, the present invention proposes a novel deep learning network for realizing tomato disease and pest target detection and classification and identification. First, the present invention constructs a YOLOX target detection model based on Swin Transformer based on the characteristics that tomato disease and pest targets are small and dense, which is used to accurately locate disease and pest targets in images. Then, the present invention constructs a rotation-invariant Fisher discriminant CNN classification model based on classic convolutional neural networks (CNN), realizes the rotation invariance of image features and improves the discrimination ability of the model, thereby improving the accuracy of disease and pest classification. The results show that: 1) When the test results are compared with traditional target detection models and classification models, the accuracy of the Swin Transformer-based YOLOX model on the tomato pest and disease test set is 7.9% and 9.5% higher than that of Faster R-CNN and SSD, respectively. The recognition accuracy of the pest and disease category of the rotation-invariant Fisher discriminant CNN model is improved by 8.7% and 5.2% compared with AlexNet and VGGNet, respectively; 2) Compared with the results of the Transformer-based target detection model DETR and the recently emerging image classification model Vision Transformer (ViT) on the tomato pest and disease test set, the detection and classification method of the present invention also has advantages, and the detection accuracy and classification accuracy of pests and diseases are improved by 3.9% and 4.3%, respectively. In addition, ablation experiments were carried out to prove the effectiveness of the model improvement of the present invention. The model constructed by the present invention outperforms other models in target detection and classification recognition of tomato pests and diseases, which helps to improve the prevention and control effect of tomato pests and diseases, and thus improve the quality and yield of tomatoes, which is of great significance to the application of computer vision in the agricultural field. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Images of tomato leaves or fruits damaged by leaf mold, early blight, gray leaf spot, whitefly, American leafminer, and cotton bollworm, as well as images of healthy tomato plants.

[0036] Figure 2 Diagram of the process of data augmentation by rotating the original training samples.

[0037] Figure 3 It is the overall workflow diagram of the present invention.

[0038] Figure 4 This is the structural diagram of the YOLOX target detection network based on Transformer.

[0039] Figure 5 Schematic diagram of the Patch Merging downsampling process.

[0040] Figure 6 This is the rotation-invariant Fisher discriminant CNN architecture diagram.

[0041] Figure 7 (a) in the figure is a confusion matrix showing the classification results of the AlexNet deep network model for the tomato pest and disease test set.

[0042] Figure 7 (b) in the figure is a confusion matrix showing the classification results of the VGGNet deep network model for the tomato pest and disease test set.

[0043] Figure 7 (c) in the figure is a confusion matrix showing the classification results of the ViT deep network model for the tomato pest and disease test set.

[0044] Figure 7 (d) in the figure is a confusion matrix showing the classification results of the rotation-invariant Fisher discriminant CNN deep network model for the tomato pest and disease test set.

[0045] Figure 8 This is a partial visualization result diagram of the detection and classification of tomato pests and diseases in the test samples using the method of the present invention. DETAILED DESCRIPTION

[0046] The specific embodiments of the present invention are further described below with reference to the accompanying drawings:

[0047] Tomatoes are one of the three most traded vegetables worldwide and hold a significant position in the global vegetable trade. With the increasing demand for tomatoes, global tomato production and cultivation are expanding, and the use of computers or artificial intelligence technologies to assist in tomato pest and disease control has become a viable strategy. Therefore, this paper proposes a novel deep learning network for target detection and classification of tomato pests and diseases. Specifically, it presents a deep learning-based method for tomato pest and disease detection and identification.

[0048] 1. Materials and methods

[0049] 1.1、Dataset:

[0050] This paper selects images of six common tomato pests and diseases, including leaf mold (Fulvia fulva), early blight (Alternaria solani), gray leaf spot (Solanum lycopersicum), whitefly (Aleurodicus dispersus), American leafminer (Liriomyza sativae) and cotton bollworm (Helicoverpa armigera) as research objects. The research dataset consists of the tomato pest and disease dataset from the Agricultural Disease and Pest Research Library (IDADP, http: / / www.icgroupcas.cn / website_bchtk / index.html) and images taken at the tomato picking base in Caoshe Village, Lianyungang City ( Figure 1 ), Figure 1 The dataset includes images of six common tomato pests and diseases, as well as images of healthy tomato plants. Rows 1 through 6 show images of tomato leaves or fruits damaged by leaf mold, early blight, gray leaf spot, whitefly, American leafminer, and cotton bollworm, respectively. The last row shows images of healthy tomato plants free of pests and diseases. The collected images with tomato pests and diseases were annotated using a tool, with the pest locations marked in each image. This yielded the first sample dataset. This dataset, when used in the object detection network, only required the object bounding boxes to be labeled as "objects" and their locations. The pest categories to which the pest locations in the tomato pest images belonged were then annotated. The second sample dataset, with the labeled pest locations and their corresponding pest categories, was used in the classification and recognition network. To ensure model stability and generalization, all images were resized to 224×224 pixels.

[0051] In order to enrich the sample data set and train the rotation invariance of the classification network, the sample needs to be rotated four times at random angles, such as Figure 2 ,After the rotation operation, the image size remains unchanged. Figure 2 In the figure, the left side is the original training sample, and the right side is the augmented sample after rotation.

[0052] The first sample data set is divided into a first sample training set and a test set. The first sample training set is subjected to rotation data augmentation processing to obtain a new first sample training set. The new first sample training set is used to train the Transformer-based YOLOX target detection network.

[0053] The second sample dataset was divided into a second sample training set and a test set. Table 1 lists the number of samples in the second sample training and test sets. 200 samples from each pest and disease category (including six specific pest and disease categories and healthy plants) were set as the test set (collection), and the remaining samples were used as the second sample training set (collection). Rotational data augmentation was performed on these samples to generate a new training set (augmentation). The sum of the collected second sample training set and the new training set generated by data augmentation was used as the new second sample training set for training the rotation-invariant Fisher discriminant CNN network.

[0054] Table 1 shows the number of training and test samples in each category of the tomato pest and disease dataset:

[0055] Pest and disease categories Training set (collection) Training set (augmented) Test set (collection) Leaf mold 256 1024 200 early blight 302 1208 200 Gray leaf spot 294 1176 200 Whitefly 283 1132 200 Liriomyza sativae 263 1052 200 cotton bollworm 327 1308 200 Healthy plants 247 988 200

[0056] 1.2 Workflow:

[0057] The workflow of this article is as follows Figure 3 As shown, the system can be divided into two parts: pest and disease target detection and pest and disease classification and recognition. First, the Transformer-based YOLOX object detection network can detect pest and disease areas and accurately locate their locations. Then, the rotation-invariant Fisher discriminant CNN network can specifically classify pests and diseases. These two networks have different primary functions, but they are used together in tomato pest and disease control to precisely control their impact and minimize losses.

[0058] 1.3. YOLOX pest and disease detection based on Transformer:

[0059] This paper's object detection network improves upon YOLOX. To address the issue of smaller targets in tomato pest and disease images, the main network, CSPDarknet, is replaced with the Swin Transformer. The attention mechanism in the Transformer comprehensively considers global feature information. The Vision Transformer (ViT) extends the Transformer model architecture to the field of computer vision. The Swin Transformer improves on ViT by using sliding windows (SW). Unlike the fixed-size sampling blocks in ViT, the Swin Transformer downsamples by a factor of 4, 8, 16, and 32 in four stages. The feature map is divided into windows, and each window is independently calculated, greatly improving computational efficiency. Furthermore, this paper replaces the original Path Aggregation Network (PANet) with a Bi-Directional Feature Pyramid Network (BiFPN) as a new feature fusion module to improve positioning accuracy. The improved Transformer-based YOLOX in this paper is mainly divided into three parts: 1. Swin Transformer is used as the backbone network to extract features of different resolutions (i.e., Swin Transformer backbone network); 2. BiFPN feature fusion module strengthens feature extraction (i.e., BiFPN enhanced feature extraction network); 3. Decoupled detection head (Decoupled Head) improves detection performance (i.e., output network); the network structure is as follows Figure 4 .

[0060] Figure 4 This is the overall network structure of YOLOX improved in this paper: (a) is the backbone network based on Swin Transformer; (b) is the bidirectional feature pyramid network (BiFPN) for feature extraction; (c) is the specific structure of the Swin Transformer module; (d) is the specific structure of the Decoupled Head.

[0061] 1.3.1. Backbone network based on Swin Transformer:

[0062] like Figure 4 As shown in (a), with an image of size [224, 224, 3] as the input of the network, the SwinTransformer backbone network is divided into four stages. Stage 1 can be divided into (1) Patch Embedding module; (2) Swin Transformer module. The specific operations are as follows:

[0063] (1) The block embedding module downsamples and transforms the image dimension through convolution. The convolution kernel size is 4×4, the stride is 4, the number of input and output channels is 3 and 96 respectively (96 is a preset dimension that is more acceptable to the Transformer). The output feature map size becomes [56, 56, 96], and the feature map shape is changed to [3136, 96] through the flatten operation.

[0064] (2) The number of Swin Transformer modules in stage 1 is 2, and the specific structure of a single module is as follows: Figure 4 As shown in (c) in the figure, the input size of the first Swin Transformer module is [3136, 96]. First, a layer normalization (LN) operation is performed on the input sequence, and the output size remains unchanged. Then, the output is reshaped to change the feature map shape back to [56, 56, 96] and enter the window-based multi-head self-attention (W-MSA) block. The W-MSA block first divides the feature map of size [56, 56, 96] into 64 windows of size [7, 7, 96] through a window partition operation. Then, a reshape operation is performed to obtain 64 sequences of size [49, 96]. Multi-head self-attention calculation is performed on the 49 elements in each window. The calculation process is as follows:

[0065] Q i =Z win W i Q , K i =Z win W i K , V i =Z win W i V (1);

[0066]

[0067] where Q i ,K i ,V i ∈R 49×32 They are the Query, Key, and Value matrices used in the multi-head self-attention calculation of each window, W i Q ,W i K ,W i V ∈R96×32 is the learnable weight matrix, Z win ∈R 49×96 Represents the sequence contained in each window after the feature map is divided into windows and reshaped, and d is Q i ,K i ,V i The dimension of the matrix, here is 32, B∈R 49×49 is the relative position offset, 49 is the number of block sequences in each window, and the number of heads used in the multi-head self-attention calculation in stage 1 is 3 (the number of heads in the four stages is 3, 6, 12, and 24 respectively). i ∈R 49×32 is the result of the self-attention calculation of the i-th head. The results of the three heads are concatenated to obtain Attention∈R 49×96 , and then the linear layer projection operation is performed to obtain the sequence Z′ of each window after the self-attention calculation win ∈R 49×96 The results of the 64 windows are concatenated to obtain an output size of [64, 49, 96] for the W-MSA block. Window Reverse is then used to restore the window form to a feature map of size [56, 56, 96]. The feature map is then flattened to [3136, 96]. After dropout and residual connections, the intermediate output is obtained. Layer normalization is performed, and a multi-layer perceptron (MLP) consisting of two linear layers and a GELU activation function is used. Finally, a residual connection is used to obtain the output of the first Swin Transformer module, of size [3136, 96].

[0068] The output of the first Swin Transformer module enters the second Swin Transformer module. The difference between the second and first Swin Transformer modules is that the W-MSA block is replaced with a sliding window-based multi-head self-attention (SW-MSA) block. The sliding window operation simultaneously shifts the 64 7×7 windows in the W-MSA block by a distance of three blocks in both the horizontal and vertical directions to enable information exchange between different windows. By setting a reasonable mask, the number of windows in the SW-MSA and W-MSA blocks is the same, achieving equivalent computational results. Other than that, the computational process is essentially the same as the first module. The input and output sizes of the Swin Transformer module remain unchanged, and the output size of stage 1 is [3136, 96].

[0069] Phase 2 consists of two steps: (1) Patch Merging module; (2) Swin Transformer module. The specific operations are as follows:

[0070] (1) The input size of stage 2 is [3136,96]. First, the input shape is converted to [56,56,96]. Then, the block merging is performed to reduce the resolution and adjust the number of channels. This is similar to the pooling operation, but without losing information. The specific operation of PatchMerging is to take a block sequence at every other position in the row and column direction of the feature map to form 4 new feature maps with half the length and width, with a size of [28,28,96]. The 4 new feature maps are spliced ​​in the direction of the number of channels. At this time, the number of channels will become 4 times the original, and the feature map size is [28,28,384] (see the schematic diagram). Figure 5 ), and then a linear mapping is used to reduce the number of channels to twice the original value, so the output size of the Patch Merging layer becomes [28,28,192], and finally the shape is transformed into [784,192]. Figure 5 Schematic diagram of the Patch Merging downsampling process.

[0071] (2) The Swin Transformer module in stage 2 is basically the same as that in stage 1, with 2 modules and input and output sizes of [784, 192].

[0072] Stages 3 and 4 are identical to stage 2. The only difference is that the Swin Transformer module in stage 3 consists of six stacked modules, while stages 1, 2, and 4 all have two modules. The Swin Transformer modules are stacked in pairs, alternating between W-MSA-based and SW-MSA-based modules. The output sizes for stages 1-4 are [3136, 96], [784, 192], [196, 384], and [49, 768], respectively.

[0073] 1.3.2、BiFPN strengthens feature extraction:

[0074] This paper modifies the feature fusion module of YOLOX into BiFPN, a weighted bidirectional feature pyramid network. Different input features have different resolutions. BiFPN adds an additional weight to each input to let the network understand the different importance of each input feature. It also adds a simple residual operation to enhance the feature representation capability, realize the bidirectional fusion of deep and shallow features from top to bottom and bottom to top, enhance the transmission of feature information between different network layers, and improve the algorithm detection accuracy and detection performance.

[0075] Stages 2, 3, and 4 of the Swin Transformer backbone network are used as effective feature layers. The outputs of these three layers are reshaped, converting [784, 192], [196, 384], and [49, 768] to [28, 28, 192], [14, 14, 384], and [7, 7, 768], respectively. These layers are then fed into the enhanced feature extraction network, BiFPN, for further feature extraction. Specifically, stage 2 of the Swin Transformer backbone network is used as the upper feature layer, stage 3 as the middle feature layer, and stage 4 as the deep feature layer. The output of the deep feature layer is reshaped to obtain a deep feature map, the output of the middle feature layer is reshaped to obtain a middle feature map, and the output of the upper feature layer is reshaped to obtain a high-level feature map. The deep, middle, and upper feature maps all serve as inputs to the BiFPN enhanced feature extraction network.

[0076] The structure of BiFPN in this paper is as follows Figure 4 As shown in (b), Upsample(2) represents 2x upsampling, which doubles the feature map resolution and halves the number of channels. Downsample(2) represents 2x downsampling, which halves the feature map resolution and doubles the number of channels. The top layer Conv(3,192,192) in the figure represents the convolution operation, where 3 represents the size of the convolution kernel of 3×3, the first 192 represents the number of input channels of the convolution operation of 192, and the second 192 represents the number of output channels. The convolution step is 1, and the filling mode is "SAME". Represents the corresponding addition of feature map elements one by one.

[0077] The upsampling path of BiFPN takes the deep low-resolution features [7,7,768] output by stage 4 as input, adjusts the resolution and number of channels by Upsample(2), and then fuses them with the input of the intermediate feature map [14,14,384]. Then, the features are extracted by the convolution operation of Conv(3,384,384) to obtain the output of the first convolution operation, which is [14,14,384]. Then, after the Upsample(2) operation, it is fused with the input of the top feature map [28,28,192]. The fused result is then convolved by Conv(3,192,192) to obtain the output of the first effective feature layer, which is [28,28,192]. For the specific process of the downsampling path, see Figure 4In (b), the output of the first effective feature layer is downsampled by a factor of 2, fused with the output of the first convolution operation and the middle-layer feature map, and then the fusion result is convolved to obtain the output of the second effective feature layer; the output of the second effective feature layer is downsampled by a factor of 2, fused with the deep-layer feature map, and then the fusion result is convolved to obtain the output of the third effective feature layer.

[0078] 1.3.3, Decoupled Head output:

[0079] Since target classification and positioning focus on different information, YOLOX uses a decoupled detection head in the final output network. The decoupled detection head uses three different branches for predicting target category, positioning, and Intersection of Union (IoU) parameters, respectively. This decouples the three and improves detection performance.

[0080] The output network includes three decoupled detection module layers, a splicing layer, and a transposition layer. The three decoupled detection module layers are simultaneously connected to the splicing layer, and the splicing layer is connected to the transposition layer. The decoupled detection module layer consists of a decoupled detection head, a splicing layer, and a shape transformation layer connected in sequence.

[0081] The decoupled detection head (Decoupled Head) includes convolutional layer, BN layer and SiLU activation function layer.

[0082] The specific structure of the decoupled detection head is as follows: Figure 4In (d) (the figure takes the input of one of the Decoupled Heads as an example), the feature map passed in by BiFPN first passes through a 1×1 convolution to convert the number of channels to 256, and then undergoes batch normalization (BN) and SiLU activation function, and then is divided into two parts, one predicting the category of the feature points and the other responsible for the position and IoU prediction of the feature points. After convolution, standardization and SiLU activation, the category prediction branch undergoes another layer of convolution, in which the convolution kernel size of the previous layer is 3×3, the step size is 1, the number of convolution kernels is 256, and the filling type is "SAME". The convolution kernel size used in the latter layer is 1×1, the step size is 1, the number of convolution kernels is the number of categories, and the filling type is "VALID". In this article, the target detection module only divides the bounding box into one category, so the number of categories is 1 and the number of output channels is 1; the process of the position and IoU prediction branch is similar to the category prediction part, except that the number of convolution kernels before the final output is different. Because YOLOX is an anchor-free detector, it directly predicts four values ​​about the position, namely the two offsets in the upper left corner of the grid, and the height and width of the prediction box, so the number of output channels of the position prediction branch is 4, and the number of output channels of the IoU prediction branch is 1. The three branch outputs of the head are concatenated in terms of the number of channels. The number of output channels of each effective feature layer becomes 6. Then, after reshape conversion, the output is transformed into a second-order tensor with the number of rows being the number of elements and the number of columns being 6. Figure 4 It can be seen that the outputs of the three Decoupled Heads become [784, 6], [196, 6], and [49, 6] after Concat and Reshape respectively. These three outputs are then concatenated and transposed (Transpose), and the final output shape is [6, 1029].

[0083] Finally, the network output is decoded, translating it into corresponding prediction boxes. The regressed location information is then mapped back to the original image, allowing the specific location of the pest to be found. After locating the pest area in the image, the specific category of the pest needs to be identified. This is achieved by using the rotation-invariant Fisher discriminant CNN model introduced in Section 1.4 to correctly classify the pest.

[0084] 1.4. Rotation-invariant Fisher discriminant CNN network:

[0085] The goal of this section is to learn a rotationally invariant Fisher discriminant CNN model to improve image feature extraction performance and enhance the accuracy of identifying tomato pests and diseases. This is achieved by optimizing the objective function based on an existing large-capacity CNN architecture, focusing on both rotational invariance and discrimination accuracy. Rotational invariance is achieved by adding a regularization constraint to the model's objective function. This constraint explicitly forces the feature representations of training samples before and after rotation to be similar, thus achieving rotational invariance. Discrimination accuracy is achieved by applying the Fisher discriminant criterion to the network features, also by adding a regularization constraint to the model's objective function to minimize intra-class scatter and maximize inter-class scatter, thereby improving discrimination capability.

[0086] 1.4.1 Network Structure

[0087] This section uses the traditional CNN network as the basic framework to build a CNN network with rotation invariance and Fisher discriminant criterion. Figure 6 As shown in the figure, the rotation-invariant Fisher discriminant CNN network consists of 5 convolutional layers, 1 three-layer fully connected layer, 1 rotation-invariant Fisher discriminant layer and 1 softmax output layer. The first 6 layers are separated by maximum pooling layers, and the activation units of all hidden layers use ReLU functions.

[0088] In the figure, Conv(3,3,64) of the first layer represents the convolution operation, the first 3 represents the convolution kernel size of 3×3, the second 3 represents the number of input channels of 3, 64 represents the number of output channels, the convolution step size is 1, and the filling mode is "SAME", so that the resolution of the image remains unchanged after convolution. The color image of size [224,224,3] is used as the input of the network. After the Conv(3,3,64) operation, the feature map size becomes [224,224,64]. The first layer is convolved twice by Conv(3,3,64) and Conv(3,64,64) to obtain an output feature map size of [224,224,64].

[0089] The pooling kernel size of all maximum pooling operations is 2×2, the stride is 2, and the padding mode is "VALID". The maximum pooling layer is used to reduce the feature map resolution, simplify the network calculation complexity and compress the feature information. After the maximum pooling operation, the feature map size of the first layer output becomes [112, 112, 64].

[0090] like Figure 6As shown in the figure, all convolution kernel sizes, convolution steps and padding modes of the first five layers are the same, and only the number of input and output channels changes. The first convolution Conv(7,512,4096) of the sixth layer processes the input of size [7,7,512] and obtains the output [1,1,4096]. It then undergoes two convolutions, Conv(1,4096,4096) and Conv(1,4096,7), to adjust the number of output channels to 7, that is, the images of 6 types of tomato diseases and pests and images of tomato plants without diseases and pests, a total of 7 categories, and the output size of the sixth layer is [1,1,7].

[0091] The seventh layer is the rotation-invariant Fisher discriminant layer, which is actually a fully connected layer of size 7 with an output size of [1, 1, 7]. The weights and biases used are trained by adding the rotation-invariant regularization constraint and the Fisher discriminant criterion objective function (Equation 17) (see Section 1.4.3 for details).

[0092] The 8th layer is the last softmax output layer. After the softmax nonlinear activation function, the output size is [1,1,7]. Softmax normalizes the output value and the output is a real number between 0 and 1. Moreover, the sum of the output values ​​of the softmax function is 1, so the output value can be regarded as the classification probability to judge the classification result.

[0093] 1.4.2. Rotation-invariant Fisher discriminant optimization objective function:

[0094] The rotation-invariant Fisher discriminant CNN model optimizes the new objective function by adding two regularization constraints. It trains the training samples before and after rotation so that they share similar features and achieve rotation invariance. At the same time, it makes the features in the network have the characteristics of small intra-class dispersion and large inter-class separation.

[0095] like Figure 6 In order to avoid overfitting and reduce training costs, all parameters (weights and biases) of the 6th layer and its preceding convolutional layers and fully connected layers are pre-trained on the ImageNet dataset and parameter adjustment is performed before transferring to the rotation-invariant Fisher discriminant CNN model. For a training sample x i ∈X RI , X RI is the set of all training samples after rotation data augmentation (that is, the second sample training set before data augmentation), let O6(x i ) is the output feature of the 6th layer, O7(x i ) is the output feature of the 7th layer rotation-invariant Fisher discriminant layer, O8(x i ) is the output of the 8th softmax classification layer, O6(xi ),O7(x i ),O8(x i )∈R 1×1×(C+1) (W7, B7) and (W8, B8) are the learnable parameters of the 7th and 8th layers respectively. Therefore, O7(x i ) and O8(x i ) can be obtained from the following equation:

[0096] O7(x i )=k(W7O6(x i )+B7) (6);

[0097]

[0098] where k(x) = max(0,x) and They are ReLU and softmax nonlinear activation functions respectively. In the training of rotation-invariant Fisher discriminant CNN, the output size of 6-layer, 7-layer, and 8-layer is [1, 1, 7].

[0099] Given the training sample X after data augmentation RI ={x i |x i ∈X∪TX} and their corresponding labels Where X represents the initial training sample, T = {T1, T2, T3, T4} represents 4 rotation operations, and TX represents the sample obtained after the rotation process. Assuming that the total number of initial training samples is N, then X RI The size is N*5, Represents the i-th sample x i The true label vector is encoded using One hot encoding and has a size of 7.

[0100] The goal of this chapter is to use input (X RI ,Y RI ) trains a rotation-invariant Fisher discriminant CNN model. In addition to requiring the model to minimize the classification error on the training dataset, it is also required that the model be rotation-invariant to any training sample set and have strong image category discrimination capabilities. To this end, a new objective function is proposed in the softmax classification layer, and the parameters are learned using the following formula, where W RI and B RI For all weights and biases of the entire network:

[0101]

[0102] Among them, λ1, λ2 and λ3 are trade-off parameters that control the relative importance of these three items.

[0103] The first term M(XRI ,Y RI ) is the softmax classification loss function, which is defined by the 7-class multinomial negative log-likelihood function. For a given training sample x i ∈X RI , which seeks the minimum misclassification error and is calculated as:

[0104]

[0105] Where <a,b> is the inner product of a and b, and 5N is the number of all training samples after data augmentation. is the image x i The category label of O8(x i )∈R 1×1×7 is the output feature of the softmax classification layer.

[0106] The second term R(X,TX) in Equation (8) is a rotation-invariant regularization constraint that is imposed on the training samples X and TX before and after rotation to ensure that they share similar features. The regularization constraint term is defined as:

[0107]

[0108] In the formula, O7(x i )∈R 1×1×7 As a training sample x i The output features of the 7th rotation-invariant Fisher discriminant layer, Represents the training sample Tx i The feature average after the rotation invariant layer, so the formula is:

[0109]

[0110] As can be seen from Equation (10), this term makes the feature of each training sample close to the average feature representation of all corresponding rotated samples. If the output value of this term is small, it means that the feature map is approximately invariant to rotation transformation.

[0111] The third term F(X RI ) is the discriminant regularization constraint of the network features. According to the Fisher discriminant criterion, X RI The intra-class scatter is minimized and the inter-class scatter is maximized. The intra-class scatter is expressed as S W (X RI ), the inter-class divergence is expressed as S B (X RI ), defined as follows:

[0112]

[0113] Among them, n j is the number of samples in the jth object class, m j Represents all samples in the jth category The average feature representation of , there are 7 categories in total, m is all samples X of all categories RI The average feature representation is obtained by the following formula:

[0114]

[0115] O7(x i ), m j and m are both 7, so S W (X RI ) and S B (X RI ) is a 7×7 matrix. Intuitively, the discriminant regularization term F(X RI ) is defined as:

[0116] F(X RI )=tr(S W (X RI ))-tr(S B (X RI )) (16);

[0117] Where tr is the trace of the matrix, which represents the sum of the diagonal elements of the matrix.

[0118] The fourth term in formula (8) Is a weight decay term that controls the weight W RI size, which helps prevent overfitting.

[0119] Substituting equations (9), (10), (12), (13) and (16) into equation (8), we get the following objective function:

[0120]

[0121] It can be seen that the defined objective function not only minimizes the classification loss, but also achieves rotation invariance through two regularization constraints, while making the learned network features more discriminative.

[0122] 1.5 Model evaluation indicators:

[0123] This paper uses precision, recall, intersection of union (IoU), and accuracy as evaluation indicators of the network model. The formula is as follows:

[0124]

[0125] For the target detection module, TP i Indicates the number of correct bounding boxes for target detection of the i-th pest and disease test sample, FP i FN represents the number of bounding boxes of the i-th pest test sample that are incorrectly located by the target detection. i Indicates the number of pest and disease bounding boxes that were missed in the i-th pest and disease sample. IoU is used to evaluate whether the pest and disease target positioning is correct. A is the predicted bounding box position of the detection target, and B is the actual bounding box position of the target. 0.5 is selected as the threshold. When the IoU between AB is greater than 0.5, the detection and positioning result is correct, otherwise it is an incorrect positioning. The accuracy is used for the classification model, where C i,i Indicates the number of samples whose true category is category i and the model predicts that they are category i, ∑ j C i,j Indicates the number of samples whose true category is the i-th category.

[0126] 2. Results:

[0127] Table 2 shows the object detection performance of the Transformer-based YOLOX object detection network constructed in this paper on a tomato pest and disease test dataset, and compares the detection results with the classic object detection networks Faster R-CNN and SSD. It can be seen that the Transformer-based YOLOX object detection network performs well in pest and disease localization, achieving an overall detection accuracy of 86.3% for six pests and diseases, and a higher recall rate (77.2%) than the other two networks (73.6% and 69.7%). Faster R-CNN draws on the attention mechanism from natural language processing (NLP) and has better detection results for small objects. However, the feature maps extracted by this network are single-layer, which is not conducive to multi-scale object detection and still has shortcomings in detecting small objects. SSD does not incorporate an attention mechanism, and its accuracy still leaves room for improvement (Precision = 76.8%) compared to Faster R-CNN (Precision = 78.4%). SSD's detection accuracy for smaller pests and diseases decreases significantly. Compared with DETR, which is also based on Transformer (Precision = 82.4%, Recall = 75.1%), this paper uses SwinTransformer as the backbone network of YOLOX, which is not only hierarchical but also takes into account both global and local information, and can achieve higher accuracy in detecting large and small objects (Precision = 86.3%, Recall = 77.2%).

[0128] Table 2: Accuracy and recall of different object detection network models on the tomato test set:

[0129]

[0130] Figure 7 (a), (b), (c), and (d) are confusion matrix diagrams showing the classification effects of the classic classification networks AlexNet, VGGNet, ViT, and the rotation-invariant Fisher discriminant CNN in this paper on the tomato pest and disease test set. The classification accuracy of the network for all test samples can be calculated from the matrix and formula 20, which are 74.1%, 77.6%, 78.5%, and 82.8%, respectively. The numbers in each column of the figure represent the proportion of each type of pest and disease test set identified as the six types of pests and diseases. The blue-purple squares represent a proportion of less than 5%, pink represents a proportion of more than 5% to less than 60%, and green represents a proportion of more than 60%. Therefore, the green squares in the three figures are concentrated on the diagonal line, representing the proportion of test samples with correct pest and disease classification. It can be seen that Figure 7 In (d), there are the most purple blocks and the least pink blocks, indicating that the rotation-invariant Fisher discriminant CNN in this paper has achieved better results and has fewer misclassifications compared to VGGNet and AlexNet. The four classification networks have the worst classification results for early blight and gray leaf spot because these two diseases have some similar characteristics. In addition, the classic CNN classification algorithm relies heavily on background information to judge the category. In the complex background of real scenes, the unoptimized VGGNet and AlexNet are more likely to make mistakes and miss classifications. In addition, due to the depth of the AlexNet network, the classification effect of the early blight and gray leaf spot is the worst. Insufficient, feature extraction effect is not good. In the test set, except for the simple classification of cotton bollworm and American leafminer, the other four pests and diseases have an error rate of about 30%. Compared with AlexNet and VGGNet, ViT uses a multi-head self-attention mechanism to better realize image classification. However, in the task of pest and disease classification, it has no obvious advantage when the differences between different types of pests and diseases are relatively small. Therefore, the classification network in this paper is deep enough and rotation invariance and Fisher discriminant optimization are performed at the same time. The number of correct classifications on each pest and disease test set is improved compared with the other three networks.

[0131] The results of the target detection and classification recognition methods in this paper are intuitively Figure 8 From the demonstration, we can see that our method can accurately detect the area where pests and diseases are located and identify the specific categories of pests and diseases. For tomatoes with dozens of pests and diseases, if we want to improve the accuracy of detection and classification in real applications, we need to consider more influencing factors. For example, the shape, color, texture and other characteristics of the same pest and disease are different in the early and late stages, and different pests and diseases may also show similar characteristics when harming plants, which will hinder the effective control of pests and diseases. At the same time, the complex natural background also has a great impact on detection and classification, such as Figure 8The second image in the first column shows that due to strong lighting in some areas of the image, individual whiteflies are not only small but also white in color. Therefore, the algorithm easily misidentifies them as background, leading to a large number of missed detections. Light intensity not only plays a role, but also the disordered interweaving of leaves and roots and the mutual obstruction of plants themselves poses a challenge. Furthermore, the distance and angle of the image can alter the shape and size of the pests.

[0132] Among the three tomato pests tested in this paper, cotton bollworms can eat tomato flowers, fruits, stems and leaves, causing rot and resulting in yield reduction. However, cotton bollworms are relatively easy to detect and classify. Figure 7 It can be seen that the detection and classification effect of cotton bollworm is the best because it has a single target, obvious features and fixed morphology. However, the detection of whitefly is the most challenging. Because it is a small target and light color, it easily blends into the background in a strong lighting environment, which brings great challenges to both target detection and classification. As shown in Table 2, the problem of missed detection of whitefly in the target detection task of Faster R-CNN network is particularly serious (Recall = 62.9%), while the Transformer-based YOLOX target detection greatly improves the recall rate (Recall = 76.3%). Both larvae and adults of the American leafminer damage tomato leaves. The larvae feed on the mesophyll on the front of the leaves, forming winding tunnels. The adults have a certain ability to fly and will suck leaf sap, damaging the early development of tomato plants. Therefore, there are two ways to identify American leafminers: one is the white curved tunnels, and the other is the flying adults attached to the plant surface. The former is difficult to detect and identify due to the complex background, while the latter is easier to detect. As can be seen in Table 2, the target detection method proposed in this paper can also effectively improve the accuracy and recall rate of detecting and classifying American leafminers (Precision = 84.5%, Recall = 78.6%).

[0133] Among tomato diseases, leaf mold can cause oval or irregular yellowish chlorotic spots to appear on the tomato leaves, and a white mold layer to form on the back of the leaves. When the disease is serious, it has different characteristics, ranging from Figure 7 As can be seen, the recognition error rates of AlexNet and VGGNet for leaf mold are 26.5% and 23.5% respectively. The misidentification results are distributed in the other five pests and diseases. To address this problem, the rotation-invariant Fisher discriminant network solves the problem of variable morphology of a pest and disease by reducing the intra-class divergence. Even if the characteristics of the same pest and disease are very different, such as Figure 8 Some leaf molds have yellow spots, while others have white mold. The rotation-invariant Fisher discriminant network can make them more closely related to the true category and improve the recognition accuracy. When gray leaf spot and early blight damage leaves, they may appear as brown or dark brown circular spots. The symptoms are very similar. Figure 8 In the 4th column, images 2 and 3 show that some of the lesions of the two diseases are similar, which leads to classification errors. Figure 7 As can be seen from the figure, the classification accuracy of models (a), (b), and (c) for these two diseases is not ideal (68% to 74%). In the case of misclassification, the two diseases are most likely to be confused with each other. Therefore, in addition to reducing intra-class dispersion, the classification network constructed in this paper also incorporates training tasks to enhance inter-class dispersion, reducing the classification error rate between diseases and insect pests with similar symptoms.

[0134] In order to study the impact of the improved backbone network and feature fusion module in the target detection network on the target positioning of tomato pests and diseases, this paper conducted an ablation experiment. The original YOLOX network and the YOLOX network with the backbone network and feature fusion module replaced were used for testing. The results are shown in Table 3. The precision of the three models decreased by 6.1%, 3.9% and 7.6% respectively. Although only replacing the backbone network achieved a higher precision, the recall rate decreased significantly, and a good balance was not achieved. The results proved the feasibility of the improved target detection network in this paper for locating tomato pests and diseases. This paper improves the classification model by adding two regularization terms to the objective function to achieve rotation invariance and Fisher discriminant. Table 4 lists the experimental results of the classification network without adding regularization and the network with rotation invariance and Fisher discriminant regularization respectively. At the same time, the performance of the model with and without rotation data enhancement on the test set is compared. The results show that optimizing the objective function for rotation invariance and Fisher discriminant can improve the accuracy of pest and disease classification. Rotation data enhancement can increase the classification accuracy of the model by 1.5% to 4.9%. Combining rotation invariance and Fisher discriminant optimization with rotation data enhancement can obtain the best classification accuracy (82.8%) in the test, giving full play to the advantages of the optimization model.

[0135] Table 3 shows the detection performance of tomato pests and diseases using the YOLOX model with different backbone networks and feature fusion modules:

[0136]

[0137]

[0138] Note: YOLOX-Swin uses Swin Transformer as the backbone network, while YOLOX-BiFPN uses BiFPN as the feature fusion module. The method proposed in this paper combines Swin Transformer and BiFPN, and is denoted as YOLOX-Swin&BiFPN.

[0139] Table 4 compares the recognition accuracy of tomato pests and diseases under different optimization strategies and different training sets:

[0140]

[0141] 3. Discussion

[0142] To address the problems of missed detection and false detection caused by small tomato pest and disease detection targets and the complex natural environment in which the targets are located, this paper uses Swin Transformer for YOLOX as the backbone network, connects the backbone network and the detection head with BiFPN, fuses multi-scale features, and constructs a rotation-invariant Fisher discriminant CNN classification method to address the classification errors caused by target angle changes, unclear inter-class feature distinctions, and diversity of similar features.

[0143] Compared with the target detection model DETR proposed in recent years, the target detection model in this paper still has advantages in the detection accuracy of tomato diseases and pests. DETR is an end-to-end target detection model based on Transformer. It encodes the entire image through a global Transformer encoder, which makes DETR more suitable for processing the detection of large-scale targets or dense targets. At the same time, it also makes it difficult for the model to accurately capture the detailed information of small and scattered targets. As shown in Table 2, for early blight and gray leaf spot, which are not very dense, the detection accuracy of DETR is not significantly improved, and DETR's detection capability is limited by the number of targets. When the number is large, DETR may not be able to accurately classify and predict bounding boxes for all objects. This paper uses Swin Transformer to extract features, which can effectively capture spatial information and contextual information at different scales, and combines BiFPN to improve the accuracy of target detection. ViT is a Transformer-based classification model. Compared to traditional convolutional neural networks, which can adapt to a variety of different image classification tasks, ViT relies entirely on the attention mechanism to calculate the relationship between input features, which provides a more intuitive understanding of the results. However, in the case of classification errors, it is difficult to accurately identify the cause of the problem, and the interpretability becomes less than ideal. When faced with the misclassification problem caused by the large intra-class dispersion, small inter-class dispersion, and variable rotation of tomato disease and insect pest features, this classification model has more advantages.

[0144] This shows that the model proposed in this paper addresses some common challenges in tomato pest and disease detection and classification and achieves good results. However, there are still unresolved difficulties in the real-world application of tomato pest and disease control, including the following two aspects:

[0145] (1) The types and manifestations of pests and diseases that exist during tomato cultivation are diverse. Tomato plants show different symptoms in the early, middle, and late stages of disease infection. Pests damage tomatoes in different ways when they are larvae and adults, and the characteristics of pests and diseases when they affect different parts of the plant vary, such as leaf wilt, fruit rot, or root deformation. At the same time, the research experiments in this paper did not take into account the situation where more than one disease or pest exists on a tomato plant at the same time and may interact with each other. Based on this, a large number of samples are needed to train the model to achieve better results, and the collection of tomato disease and pest image samples is also a difficult problem.

[0146] (2) In order to improve accuracy, this paper divides detection and classification into two models, so the training samples need to be labeled twice and the models need to be trained separately. This requires more time and effort when faced with more types of pests and diseases and a large number of samples in field tomato cultivation management.

[0147] Future work will expand the tomato pest and disease dataset by further subdividing the categories to include different stages of the same disease, different forms of the pest, and increasing sample diversity by adding images captured under different lighting conditions and from varying shooting angles. Collecting images of multiple common pests and diseases that harm tomato plants will prepare for subsequent in-depth research. Further network optimization will be pursued, with research on improved object segmentation techniques to help the model better detect and distinguish objects. Adjustments to the training dataset, model architecture, and parameter settings will be made to improve the model's interpretability and enhance detection and classification performance for more complex datasets. Effective methods will be developed to ensure high accuracy in both localization and classification while integrating detection and classification models to simplify the processing process.

[0148] 4. Conclusion

[0149] This paper proposes a method for tomato pest and disease target detection and classification recognition that combines target detection and classification recognition models. In response to the common problems in tomato pest and disease detection and recognition, a Transformer-based YOLOX target detection model and a rotation-invariant Fisher discriminant CNN classification model are proposed, and rotation data enhancement is performed on the training samples. This method improves the accuracy of pest and disease target detection, reduces missed detections, and also ensures a high recognition ability for tomato pest and disease types. Test results show that the Transformer-based YOLOX object detection achieves excellent results in pest and disease localization. On a tomato pest and disease test set, the overall detection precision reached 86.3% and the recall rate increased to 77.2%. Compared with traditional classification and recognition CNN models such as AlexNet and VGGNet, the rotation-invariant Fisher discriminant CNN model also significantly improves the classification accuracy of tomato pests and diseases, increasing the classification accuracy of six pests and diseases (leaf mold, early blight, gray leaf spot, whitefly, American leafminer, and cotton bollworm) to 83%, 78.5%, 76.5%, 79.5%, 88.5%, and 91.5%, respectively. Future research will focus on developing more effective models suitable for real-world pest and disease control applications, enriching datasets while also optimizing the network. This paper separates object detection and classification into two models. While highlighting the advantages of each model, the processing speed is not significantly superior, so the speed of detection and classification will be a key focus for future research and improvement.

[0150] The protection scope of the present invention includes but is not limited to the above embodiments. The protection scope of the present invention is based on the claims. Any replacement, deformation, and improvement of the technology that can be easily thought of by those skilled in the art fall within the protection scope of the present invention.

Claims

1. A method for detecting and identifying tomato pests and diseases based on deep learning, characterized in that: The following steps are involved: (1) Collect multiple tomato pest and disease images and mark the location of the pests in each tomato pest and disease image. The original tomato pest and disease images and the tomato pest and disease images with marked pest and disease locations are used as the first sample training set; (2) marking the pest and disease category to which the pest and disease location in the tomato pest and disease image obtained in step (1) belongs, and using the tomato pest and disease image with the pest and disease location marked and the pest and disease category to which it belongs as the second sample training set; Collect multiple images of healthy tomato plants and use the images of healthy tomato plants and their health categories as the second sample training set; (3) Perform data enhancement on the first sample training set and the second sample training set respectively to obtain a new first sample training set and a new second sample training set; (4) Using the new first sample training set to train the Transformer-based YOLOX target detection network, a trained pest and disease target detection model is obtained; (5) Using the new second sample training set to train the rotation-invariant Fisher discriminant CNN network, a trained pest and disease classification and recognition model is obtained; (6) Inputting the tomato image to be detected into the pest and disease target detection model to locate the location of the pest and disease in the tomato image; (7) The image in which the location of the pests and diseases in the tomato image is located is input into the pest and disease classification and recognition model, thereby identifying the category of the pests and diseases in the tomato image.

2. The tomato pest and disease detection and identification method based on deep learning according to claim 1 is characterized in that: The data enhancement is to perform 4 random angle rotation operations on the sample.

3. The tomato pest and disease detection and identification method based on deep learning according to claim 1 is characterized in that: The Transformer-based YOLOX target detection network includes a Swin Transformer backbone network, a BiFPN enhanced feature extraction network, and an output network.

4. The tomato pest and disease detection and identification method based on deep learning according to claim 3 is characterized in that: The Swin Transformer backbone network includes stage one, stage two, stage three and stage four; The first stage includes a block embedding module and two Swin Transformer modules for downsampling and dimensionality transformation of the input image through convolution; the second stage includes a block merging module and two Swin Transformer modules for performing shape transformation and block merging on the input image; the third stage includes a block merging module and six Swin Transformer modules for performing shape transformation and block merging on the input image; the fourth stage includes a block merging module and two Swin Transformer modules for performing shape transformation and block merging on the input image.

5. The tomato pest and disease detection and identification method based on deep learning according to claim 4 is characterized in that: The BiFPN enhanced feature extraction network includes an upsampling path and a downsampling path; The second stage of the Swin Transformer backbone network is used as the upper feature layer, the third stage as the middle feature layer, and the fourth stage as the deep feature layer. The output of the deep feature layer is reshaped to obtain the deep feature map, the output of the middle feature layer is reshaped to obtain the middle feature map, and the output of the upper feature layer is reshaped to obtain the upper feature map. The deep feature map, middle feature map and upper feature map are all inputs of the BiFPN enhanced feature extraction network; The upsampling path of the BiFPN enhanced feature extraction network is as follows: the deep feature map is fused with the middle feature map after a 2-fold upsampling operation, and then the features are extracted through a convolution operation to obtain the output of the first convolution operation. The output of the first convolution operation is then upsampled by 2 times and fused with the upper feature map. The fused result is then convolved to obtain the output of the first effective feature layer. The downsampling path of the BiFPN enhanced feature extraction network is as follows: the output of the first effective feature layer is downsampled by a factor of 2, fused with the output of the first convolution operation and the middle-layer feature map, and then the fused result is convolved to obtain the output of the second effective feature layer; The output of the second effective feature layer is downsampled by a factor of 2 and then fused with the deep feature map. The fused result is then convolved to obtain the output of the third effective feature layer.

6. The method for detecting and identifying tomato pests and diseases based on deep learning according to claim 5, characterized in that: The output network includes three decoupling detection module layers, a splicing layer, and a transposition layer. The three decoupling detection module layers are simultaneously connected to the splicing layer, and the splicing layer is connected to the transposition layer. The decoupling detection module layer is composed of a decoupling detection head, a splicing layer, and a shape transformation layer connected in sequence. The decoupled detection head consists of a convolutional layer, a BN layer, and a SiLU activation function layer.

7. The tomato pest and disease detection and identification method based on deep learning according to claim 1, characterized in that: The rotation-invariant Fisher discriminant CNN network includes 5 convolutional layers, a fully connected layer, a rotation-invariant Fisher discriminant layer and a softmax classification layer; The first convolutional layer includes two convolution operations, the second convolutional layer includes two convolution operations, the third convolutional layer includes three convolution operations, the fourth convolutional layer includes three convolution operations, the fifth convolutional layer includes three convolution operations, and the fully connected layer includes three convolution operations; there are maximum pooling layers between the first and second convolutional layers, between the second and third convolutional layers, between the third and fourth convolutional layers, between the fourth and fifth convolutional layers, and between the fifth convolutional layer and the fully connected layer.

8. The method for detecting and identifying tomato pests and diseases based on deep learning according to claim 7, characterized in that: The objective function of the rotation-invariant Fisher discriminant CNN network is: Among them, W RI represents all weights of the rotation-invariant Fisher discriminant CNN network, B RI Represents all biases of the rotation-invariant Fisher discriminant CNN network, X RI ={x i |x i ∈X∪TX}, represents the new second sample training set after data enhancement, where X represents the second sample training set before data enhancement, T = {T1, T2, T3, T4} represents 4 rotation operations, TX represents the sample data obtained after all samples in X are rotated 4 times, and the total number of the second sample training set before data enhancement is N, X RI The number is N*5; λ1, λ2 and λ3 represent trade-off parameters, and <a,b> represents the inner product of a and b. Represents the training sample x i The category label of O8(x i ) represents the training sample x i The output features of the softmax classification layer, O7(x i ) represents the training sample x i The output features of the rotation-invariant Fisher discriminant layer, Represents the training sample x i The average output feature of the sample obtained after 4 rotation operations is the rotation-invariant Fisher discriminant layer, n j is the number of samples in the jth category, m j Represents all samples in the jth category The average feature representation of , there are 7 categories in total, m is all samples X of all categories RI The average feature representation of , tr is the trace of the matrix, which represents the sum of the diagonal elements of the matrix.

Images