Intelligent detection and identification system and method for mulberry diseases and insect pests
Through the intelligent detection and identification system of mulberry pests and diseases, the improved YOLOv5 model and multi-scale feature extraction technology are used to solve the problems of insufficient data and complex background interference in mulberry pest detection, and improve the detection accuracy and efficiency, especially in small-target pest detection. It significantly improves accuracy.
Patent Information
- Application Number
- CN202411007953.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
AI Technical Summary
The data samples are limited in mulberry pest detection, traditional detection methods are prone to missed detection, and complex background interference leads to low detection accuracy, especially low detection accuracy of small target pests.
A smart detection and identification system for mulberry diseases and pests was designed, including video acquisition module, image processing module, pest feature extraction module, disease feature extraction module and detection model module. The improved YOLOv5 model is adopted, and through the SCER module, C3-ECA module, Technical means such as RFEM module, ssFPN unit and ODConv unit extract and fuse multi-scale features to enhance the detection ability of small targets.
It improves the accuracy and efficiency of mulberry pest detection, especially in complex backgrounds, can work stably, enhances the robustness and generalization ability of the model, and achieves accurate identification and positioning of small-target pests.
Smart Images

Figure CN118968295A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of image recognition models, and in particular to an intelligent detection and recognition system and method for mulberry tree pests and diseases. Background Art
[0002] In the field of mulberry pest detection and prevention, with the development of science and technology, the use of artificial intelligence technology for mulberry pest detection and prevention has gradually attracted attention. However, there are many problems in practical application.
[0003] At present, the data set of pests and diseases in the field of mulberry trees is very limited, resulting in a lack of sufficient data samples for model training. At the same time, many mulberry pests appear as small targets in the image due to their small size or shooting distance, which makes traditional detection methods prone to false detection and missed detection. In addition, mulberry diseases usually occur in real production scenes, and complex backgrounds such as light and weather can cause serious interference to disease detection and reduce detection accuracy. In terms of detection of small target pests in mulberry trees, the useful features that can be extracted from small target pests themselves are limited, and as the number of network layers increases, their features are gradually lost during the downsampling process, resulting in low detection accuracy. For mulberry disease detection, complex and changeable background conditions often lead to errors in detecting leaf diseases, which in turn affects the accuracy of prevention and control measures.
[0004] Therefore, in order to solve the above problems and improve the accuracy and efficiency of mulberry tree disease and insect pest detection, the present invention proposes a mulberry tree disease and insect pest intelligent detection and identification system and method. Summary of the invention
[0005] The present invention provides a mulberry tree disease and insect pest intelligent detection and identification system and method, which can improve the accuracy and efficiency of mulberry tree disease and insect pest detection.
[0006] In order to solve the above technical problems, this application provides the following technical solutions:
[0007] A mulberry tree disease and insect pest intelligent detection and identification system, comprising:
[0008] Video acquisition module, used to collect image data of mulberry tree diseases and pests in real time;
[0009] An image processing module, used for preprocessing the collected image data;
[0010] The pest feature extraction module is used to extract the features of pests from the preprocessed images;
[0011] A disease feature extraction module is used to extract disease features from the preprocessed image;
[0012] Detection model module, used to identify pests and diseases based on the extracted features;
[0013] The data management module is used to store and manage the test results.
[0014] Furthermore, the pest feature extraction module includes a Backbone unit, a Neck unit and a Head unit; the Backbone unit is used to extract image features, including a SCER module, and the SCER module includes a Focus+SPD submodule, a C3-ECA submodule and an RFEM submodule; the Neck unit is used to fuse features, including an ssFPN unit and an ODConv unit, and the ssFPN unit is used to regard each layer of feature maps in the feature pyramid as each frame in the video for Concat splicing, and then input into the 3D convolution; the ODConv unit is used to introduce full-dimensional dynamic convolution, and learn from four dimensions through a multi-dimensional attention mechanism; the Head unit is used to detect and identify pest types.
[0015] Furthermore, the Focus+SPD submodule is used to replace the convolutional layer of the 0th layer of the backbone network with the Focus structure, and add an SPD module after each convolutional layer.
[0016] Furthermore, the C3-ECA submodule adds an ECA attention mechanism to the C3 module to learn the channel attention of each convolutional block.
[0017] Furthermore, the RFEM submodule is added after the SPPF spatial pyramid pooling layer, and three dilated convolutions with different dilation rates are used to extract multi-scale information.
[0018] Furthermore, the ssFPN unit adjusts the feature maps of each layer of the feature pyramid to a uniform size through operations such as convolution upsampling, and uses 3D convolution to extract scale sequence features.
[0019] Furthermore, the ODConv unit is added after the two C3 modules in the top-down path, introducing a multi-dimensional attention mechanism.
[0020] Furthermore, the disease feature extraction module includes a C3-Res2block submodule and a ContextAggregation submodule; the Context Aggregation submodule includes an SE unit; the C3-Res2block submodule is used to replace the 3*3 filters of c channels with a group of smaller filters, and then group the input features. The filter first learns information from the first group of feature maps, and then the output features are input into the group of filters together with the next group of feature maps again, and this is repeated until all group features are processed, and then all feature maps are concat-connected, and then input into the 1*1 filter for fusion, and finally input into the SE unit to adaptively adjust the channel; wherein the filter group is connected in a residual manner; the ContextAggregation submodule is used to fuse the spatial context at the pixel level inspired by the self-attention mechanism:
[0021]
[0022] Here P i and Q i They represent the input and output feature maps of the i-th layer in the feature pyramid, and each feature map contains N i pixels, j,m∈{1,N i} represents the index of each pixel, w k and w v is the linear projection matrix used to project the feature map. Formula 1 replaces the matrix multiplication of query and key by linear projection; a linear projection matrix with P is applied. i and Q i The updated weight matrix of the same shape is used to balance the degree to which each pixel aggregates the spatial context. This matrix can also be simply derived from P i Perform linear projection and generate through Sigmoid activation;
[0023]
[0024] Again, j∈{1,N_i} is the matrix index and the output of the Sigmoid function is regarded as the proportion of information aggregated from the global context.
[0025] A method for intelligent detection and identification of mulberry tree diseases and insect pests comprises the following steps:
[0026] Step 1: Collect image data of mulberry tree diseases and insect pests in real time through the video acquisition module;
[0027] Step 2: Preprocess the collected image data through an image processing module, including but not limited to denoising, contrast enhancement, etc.;
[0028] Step 3: Extract the features of pests and diseases in the preprocessed image through the feature extraction module;
[0029] Step 4: Identify the extracted features through the detection model module to determine the type and location of pests and diseases;
[0030] Step 5: Store and manage the test results through the data management module.
[0031] The principle and beneficial effects of the scheme are as follows: the intelligent detection and identification system for mulberry pests and diseases of the present invention adopts a modular design, including a video acquisition module, an image processing module, a feature extraction module, a detection model module and a data management module. Each module works in coordination to achieve real-time detection and identification of mulberry pests and diseases.
[0032] The feature extraction module is the core of the system, including the pest feature extraction module and the disease feature extraction module. The pest feature extraction module is further subdivided into the Backbone unit, the Neck unit and the Head unit. The SCER module includes the Focus+SPD submodule, the C3-ECA submodule and the RFEM submodule. The Focus+SPD submodule replaces the convolution layer of the 0th layer of the backbone network with the Focus structure and adds the SPD module after each convolution layer to enhance the ability to extract small target features.
[0033] The ECA attention mechanism is added to the C3 module to learn the channel attention of each convolution block and improve the expressiveness of features. It is added after the SPPF spatial pyramid pooling layer and uses three different dilated convolutions with different dilated rates to extract multi-scale information and enhance the detection capability of small targets.
[0034] The ssFPN unit concatenates the feature maps of each layer in the feature pyramid as each frame in the video, and then inputs them into the 3D convolution to enhance the learning ability of small target features.
[0035] The ODConv unit introduces full-dimensional dynamic convolution and learns from four dimensions through a multi-dimensional attention mechanism to further improve the feature extraction capability of small objects.
[0036] The Head unit is used to detect and identify the type of pests and to perform the final classification and positioning of the extracted features.
[0037] The C3-Res2block submodule replaces the 3*3 filter with a smaller filter, and improves the deep mining capability of features through grouping and residual connection. The Context Aggregation submodule fuses the spatial context at the pixel level through the self-attention mechanism to enhance the global expression capability of features. The detection model module uses the improved YOLOv5 model and uses the extracted features to detect and identify pests and diseases.
[0038] By improving the YOLOv5 model, especially the detection accuracy of small target pests, the system can more accurately identify mulberry pests and diseases, and improve the accuracy of detection. By introducing the RFEM module and ssFPN unit, the system can extract multi-scale features and enhance the detection ability of targets of different sizes.
[0039] Through the ECA attention mechanism and Context Aggregation sub-module, the system can better learn the channel attention and spatial context information of features, improving the expressiveness of features.
[0040] The system uses real-time video acquisition and fast image processing technology to achieve real-time detection of mulberry pests and diseases, meeting the timeliness requirements of agricultural production. Through data enhancement and multi-dimensional attention mechanisms, the system can also work stably in complex backgrounds, improving the robustness and generalization ability of the model.
[0041] The design of the data management module makes it easy to store and manage the test results, facilitates subsequent data analysis and decision support, and improves the practicality and operational convenience of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is the YOLOv5 benchmark network structure diagram.
[0043] Figure 2 (Left) The height-to-width ratio of the target in the image in the YOLOv5 training results; (Right) A diagram of the confusion matrix visualization of the classification effect of the YOLOv5 model on different types of pests.
[0044] Figure 3 Schematic diagram of the improved YOLOv5 network structure.
[0045] Figure 4 This is a schematic diagram of the SPD module.
[0046] Figure 5 Schematic diagram of the ECA attention mechanism.
[0047] Figure 6 Schematic diagram of the RFEM module.
[0048] Figure 7 Schematic diagram of the scale sequence module.
[0049] Figure 8 Schematic diagram of the ODConv module.
[0050] Fig. 9 This is a schematic diagram of the PR curve.
[0051] Fig.10Schematic diagram of the feature heat map comparison of the baseline YOLOv5 network and the improved network learning small target features.
[0052] Fig.11 Schematic diagram of the overall architecture of the improved disease detection model.
[0053] Fig.12 Figure 2 shows the data augmentation method.
[0054] Fig.13 This is the effect diagram of the random data enhancer.
[0055] Fig.14 This is a schematic diagram of the Res2net module.
[0056] Fig.15 This is a schematic diagram of the ContextAggregation module.
[0057] Fig.16 Schematic diagram of IoU intersection-over-union ratio.
[0058] Fig.17 This is a comparison chart of model detection performance under different IoU.
[0059] Fig.18 Schematic diagram of the mulberry disease model comparison experiment.
[0060] Fig.19 This is a heat map of mulberry disease characteristics. DETAILED DESCRIPTION
[0061] The following is further described in detail through specific implementation methods:
[0062] A mulberry tree disease and insect pest intelligent detection and identification system, comprising:
[0063] Video acquisition module, used to collect image data of mulberry tree diseases and pests in real time;
[0064] An image processing module, used for preprocessing the collected image data;
[0065] The pest feature extraction module is used to extract the features of pests from the preprocessed images;
[0066] Disease feature extraction module, used to extract disease features from preprocessed images
[0067] Detection model module, used to identify pests and diseases based on the extracted features;
[0068] The data management module is used to store and manage the test results.
[0069] When used specifically: the pest feature extraction module includes Backbone unit, Neck unit and Head unit. Backbone unit is used to extract image features, including SCER module.
[0070] The SCER module includes the Focus+SPD submodule, the C3-ECA submodule, and the RFEM submodule. The Focus+SPD submodule is used to replace the convolutional layer of the 0th layer of the backbone network with the Focus structure and add an SPD module after each convolutional layer.
[0071] The C3-ECA submodule adds the ECA attention mechanism to the C3 module to learn the channel attention of each convolution block. The RFEM submodule is added after the SPPF spatial pyramid pooling layer and uses three different dilated convolutions with different dilated rates to extract multi-scale information. The Neck unit is used to fuse features, including the ssFPN unit and the ODConv unit.
[0072] The ssFPN unit treats each layer of feature maps in the feature pyramid as each frame in the video, concatenates them, and then inputs them into the 3D convolution. The ODConv unit introduces full-dimensional dynamic convolution and learns from four dimensions through a multi-dimensional attention mechanism. The Head unit is finally used to detect and identify pest types.
[0073] The disease feature extraction module includes the C3-Res2block submodule and the Context Aggregation submodule. The C3-Res2block submodule replaces the 3*3 filters of c channels with a group of smaller filters, and then groups the input features. The filter first learns information from the first group of feature maps, and then inputs the output features together with the next group of feature maps into the group of filters again. This is repeated until all group features are processed, and then all feature maps are concat-connected, and then input into the 1*1 filter for fusion, and finally input into the SE unit to adaptively adjust the channels. The filter groups are connected in a residual manner.
[0074] The Context Aggregation submodule is inspired by the self-attention mechanism and can integrate the spatial context at the pixel level.
[0075]
[0076] Here P i and Q i They represent the input and output feature maps of the i-th layer in the feature pyramid, and each feature map contains N i pixels, j,m∈{1,N i} represents the index of each pixel, w k and wv is the linear projection matrix used to project the feature map. Formula 1 replaces the matrix multiplication of query and key by linear projection; a linear projection matrix with P is applied. i and Q i The updated weight matrix of the same shape is used to balance the degree to which each pixel aggregates the spatial context. This matrix can also be simply derived from P i Perform linear projection and generate through Sigmoid activation;
[0077]
[0078] Again, j∈{1,N_i} is the matrix index and the output of the Sigmoid function is regarded as the proportion of information aggregated from the global context.
[0079] The detection model module uses the improved YOLOv5 model, which can identify the extracted features and determine the type and location of pests and diseases through training. The data management module is responsible for storing and managing the detection results to facilitate subsequent analysis and processing.
[0080] In specific use: the YOLOv5 model is selected as the benchmark model. Because it has rich practical experience in multiple application fields, it has stronger stability and practicality than other models. Its network structure is as follows Figure 1 After training the constructed mulberry pest dataset using the YOLOv5 benchmark model, it was found that many pest targets were small in size and had a low detection rate, such as Figure 2 As shown in the figure, the left picture represents the height-width ratio of the target in the image. It can be seen from the figure that pixels with a ratio of less than 0.1 are darker, indicating that quite a number of pest targets only occupy a small part of the image. These targets can be called small targets. The right picture represents a visualization of the confusion matrix, which is used to show the classification effect of the YOLOv5 model on different types of pests. The rows of the matrix represent the real categories, the columns represent the predicted categories, and the values on the non-diagonal lines represent the number of samples predicted incorrectly by the model. By observing the values, it can be found that many categories are incorrectly predicted as background, such as red spiders and green stink bugs, which are all small target pests.
[0081] In order to further improve the accuracy of mulberry pest detection, we need to start from the perspective of improving the accuracy of small target pest detection. By reading and studying a large number of relevant literature, we can understand the fundamental reasons for the low accuracy of small targets. We also improved the YOLOv5 model and finally successfully improved the accuracy of mulberry pest detection by 3.9%. The improved YOLOv5 model is as follows Figure 3 shown.
[0082] Next, the overall framework of the improved YOLOv5 model based on the low accuracy of small target pest detection is introduced. According to its structure, it can be divided into three parts: Backbone (feature extraction), Neck (feature fusion) and Head (detection head). The improvement work is mainly concentrated in the Backbone and Neck parts.
[0083] 1. Backbone (feature extraction)
[0084] Backbone refers to the backbone network of YOLOv5, which is mainly used to extract features from images for use by the subsequent networks. The improvement work in this chapter still uses Darknet-53 as the main feature extraction network, and then designs a SCER module, which replaces the convolution layer of the backbone network at layer 0 with a Focus structure, and adds an SPD module after each convolution layer to better retain the transmission of small target feature information in the network and prevent useful features from disappearing. Then, the C3 module used by the YOLOv5 backbone network to extract image features is improved, and the ECA attention mechanism is added to enhance the features of the channel dimension, so as to better emphasize the useful features of small targets. Then, a multi-scale feature extraction module RFEM is added after the last layer of spatial pyramid pooling in the backbone network. This module uses three different dilated convolutions with different dilated rates to enhance the small target feature expression ability of the neural network.
[0085] 2. Neck (feature fusion)
[0086] The Neck feature fusion part mainly uses the feature pyramid structure to fuse shallow graphic features and deep semantic features at multiple scales, thereby obtaining more complete features and passing them to the Head prediction head. The inspiration for the improvement work in this chapter comes from the video recognition task. Each layer of feature maps in the feature pyramid is regarded as each frame in the video for concat splicing, and then input into the 3D convolution, so as to establish the connection between feature maps of different layers and enhance the learning ability of small target features. On this basis, full-dimensional dynamic convolution is introduced, and learning from four dimensions is carried out through the multi-dimensional attention mechanism to further improve the feature extraction ability of small targets.
[0087] Further explanation of the improvement work content:
[0088] 1. Backbone part - SCER module
[0089] The SCER module mainly consists of three parts: Focus+SPD module, C3-ECA module and RFEM module.
[0090] (1) Focus+SPD module
[0091] In the process of feature extraction, the convolutional neural network will lose fine-grained information of small targets due to the existence of cross-convolutional layers and pooling layers. On the one hand, in the pooling process, only the maximum value or average value is selected. Although such an operation can reduce the size of the feature map, some target details will be lost. The cross-convolutional layer will also reduce the resolution of the feature map, making it more difficult for the network to accurately capture the feature information of small targets. In order to deal with this problem, the convolutional layer in YOLOv5 is replaced by SPD-Con. SPD-Conv mainly consists of two parts: non-strided convolutional layer and space-to-depth layer. Figure 4 shown.
[0092] The first half of the method splits the feature map from the spatial dimension, then concatenates it according to the channel dimension, and then combines it with a non-strided convolutional layer with a step size of 1, so that more detailed information can be transmitted through the network without excessively reducing the size of the feature map. This can well preserve the important detail features of small targets. Then, by replacing SPD-Conv at different positions of YOLOv5, it is found that replacing only the first three Conv convolutional layers of Backbone can achieve an ideal balance between the number of parameters and detection accuracy. The reason is that these layers are located in the backbone network and are mainly responsible for extracting features from the middle and lower layers. Replacing only the first three convolutional layers is more conducive to maintaining the extraction of multi-scale features. If all are replaced, the depth of the model will increase, resulting in instability in training. The introduction of the Focus structure in the early stage of the network processes the input image, thereby reducing the amount of calculation while maintaining sensitivity to small targets.
[0093] (2) C3-ECA module
[0094] In order to improve the ability to extract small target features, the C3 module used to extract image features in the network needs to be improved. Therefore, the ECA attention mechanism is added to the C3 modules in the first four layers of the network, such as Figure 5 As shown in the figure, the channel attention of each convolution block is learned. At the same time, the attention mechanism avoids dimensionality reduction and can dynamically adjust the size of the convolution kernel according to the importance of each channel. This allows the model to more adaptively focus on important features while suppressing unimportant features, so as to better capture the relevant features of small objects.
[0095] (3) RFEM module
[0096] The receptive field sizes of dilated convolutions with different dilation rates on the input features are different. Smaller dilation rates correspond to smaller receptive fields, which are used to capture local features and detail information, and are more suitable for processing small-scale targets or fine-grained features. Larger dilation rates correspond to larger receptive fields, which are used to capture more extensive contextual information and are suitable for processing large-scale targets or global features. A medium dilation rate provides a medium receptive field size, which can compromise between local and global, and is suitable for processing medium-scale targets or features. This feature of dilated convolution is consistent with the characteristics of constructing the mulberry pest dataset (there are both large, medium and small target pests), so the RFEM structure is added after the SPPF spatial pyramid pooling layer of the backbone network. This module uses three dilated convolutions with different dilation rates to extract multi-scale information, share weights between different branches, and use residual connections to prevent gradient explosion. Finally, the features of the four branches are added and output, as shown in Figure 6 As shown in the figure, using convolution kernels with different void rates to extract multi-scale information can effectively enhance the feature expression ability of deep neural networks and achieve the effect of enhancing the semantic information of small objects.
[0097] 2. Neck part - SSOD-FPN
[0098] After improving the Backbone feature extraction part, the model can not only ensure that the small target features are retained to a certain extent during the neural network transmission process, but also enhance the ability to express the detailed features of small targets. Next, we improved the Neck feature fusion part of YOLOv5 and proposed a custom feature pyramid SSOD-FPN to further enhance the feature pyramid's ability to extract small target features. The improvement work mainly consists of two parts:
[0099] (1)ssFPN
[0100] The multi-scale representation of features is often expressed through scale space. The scale space is usually defined along the scale of the image, which represents the various scale ranges that the target can have. The generation of scale space usually uses Gaussian filters to blur the image instead of directly adjusting the image size. Neural network models often learn scale-invariant features from the multi-scale features of the target. The specific implementation method is to use feature pyramid networks to process multi-scale targets, but the general feature pyramids are all directly spliced and other operations to perform feature fusion, which is limited in alleviating the semantic gap between the feature maps of each layer of the pyramid. Semantic information is very important for small target detection. Inspired by 3D convolution in traditional video recognition tasks, the difference between 3D convolution and traditional 2D convolution is that 3D convolution introduces the time dimension in the convolution kernel, so it can capture both spatial and temporal features. In YOLOv5, the feature maps of different layers of the pyramid of the general view are regarded as each frame in the video through 3D convolution, and the time axis of the video frame is regarded as the horizontal axis of the general view. Then, the unique features obtained by splicing the pyramids of each layer are defined as scale sequence features after 3D convolution. The feature can include scale correlation across feature pyramids. Scale sequence characteristics (S 2 ) is defined as:
[0101]
[0102] in It is a scale sequence module obtained by 3D convolution, which regards the horizontal axis of the feature map as the time axis of the video frame.
[0103] G=(time·width·height·channel) (3-2)
[0104] Used to extract scale sequence features from views (S 2 ),like Figure 7 shown.
[0105] Since small targets are usually detected in high-resolution feature maps, such as P3. Therefore, in the scale sequence module, the feature maps of each layer of the pyramid are adjusted to a unified P_3 size resolution through operations such as convolution upsampling, and Unsqueeze is used to add a Level dimension to each feature map, and then concatenated with Concat and input into the 3D convolution block. The 3D convolution block mainly consists of three parts: 3D convolution, 3D batch normalization, and Leaky ReLu activation function. The scale sequence features after 3D convolution are combined with the P_3 feature map and input into the Head detection head for detection. The detection head channel is:
[0106]
[0107] in It is the result of concatenating the scale sequence features and P3 features.
[0108] (2)ODConv
[0109] After further study of convolution, it was found that conventional convolution has only one static convolution kernel and is independent of the input sample, which limits the expressive power of the model. Therefore, after learning other convolutions, it was found that dynamic convolution can improve the feature representation ability without increasing the depth or width of the network. The basic idea is to use attention to dynamically aggregate multiple parallel convolution blocks instead of using a single convolution kernel, so as to adaptively adjust the weight of each convolution kernel according to the input image. This attention mechanism improves the representation ability of the convolution kernel by nonlinearly superimposing the convolution kernel, so it is planned to use dynamic convolution instead of conventional convolution. However, after in-depth understanding of dynamic convolution, it was found that general dynamic convolution such as CondConv and DyConv only uses attention scalar from one dimension, while convolution kernels usually have four dimensions: spatial kernel size, number of input channels, number of output channels, and number of convolution kernels. The neglect of other dimensions has led to only rough exploration of the convolution kernel space, which has limited performance gains for the network. However, ODConv uses a parallel strategy to introduce a multi-dimensional attention mechanism to gradually learn more flexible attention for the four dimensions of convolution, such as Figure 8 shown.
[0110] This operation makes the input features different in all dimensions, so as to better capture rich contextual information and improve the feature extraction ability of the convolution kernel. The form of ODCconv is as follows:
[0111] y=(α w1 ⊙α f1 ⊙α c1 ⊙α s1 ⊙W1+…+α wn ⊙α fn ⊙α cn ⊙α sn ⊙Wn n )*x (3-4)
[0112] where α wi Represents the convolution kernel W i The attention scalar, and α si , α ci , α fi Represents the attention introduced from the spatial dimension, input channel dimension, and output channel dimension. wi Different values are assigned to multiple overall convolution kernels, and α fi and α ci Different attention values are assigned to the convolution filters of different input and output channels, α siThe convolution parameters at the S×S spatial domain position are assigned attention values. These four attentions are passed through the multi-head attention module π i (x) is obtained. Therefore, the improvement work in this section is to add ODConv after the two C3 modules in the top-down path of Neck in YOLOv5, and combine it with the scale sequence module to propose a new feature pyramid SSOD-FPN to establish the connection between feature maps and improve the feature extraction ability of convolution for small targets.
[0113] To further illustrate the effect, relevant experiments are designed for comparison:
[0114] 1. Experimental settings and evaluation indicators
[0115] (1) Experimental setup
[0116] The detection model in this chapter uses a self-built mulberry pest dataset to conduct deep learning experiments on the Pytorch platform, and uses the MMdetection platform for comparative experiments. During the experiment, the hardware environment used was TITAN Xp 12G video memory, and the software environment was Pytorch: 1.4.0, Cuda: 10.1. The experiment adopted a single GPU training method, the initial learning rate was set to 0.01, and the rest of the hyperparameters were based on the initial default parameters of the YOLOv5 network. In order to better compare the performance gain of the improved YOLOv5 model, training was performed without using pre-training parameters. During the entire training process, the number of Epoch iterations was 200 and the BatchSize was 16.
[0117] (2) Evaluation indicators
[0118] In the field of object detection, precision, recall, and mean average precision (mAP) are often used to judge the accuracy of a model. TP means that positive samples are judged as true and the judgment is correct, and FP means that negative samples are judged as negative and the judgment is correct.
[0119]
[0120] Therefore, Presion indicates the proportion of positive samples predicted correctly by the model to all samples predicted as positive.
[0121]
[0122] It represents the proportion of positive samples predicted by the model to all positive samples, where FN represents the judgment of negative samples as positive, but the judgment is wrong. Fig. 9 shown.
[0123] AP is the area under the PR curve, which represents the average accuracy of a certain category of detection.
[0124]
[0125] mAP represents the average AP of each category, as shown by the blue line in the figure above.
[0126]
[0127] Another concept involved is the intersection over union (IoU), which represents the ratio of the intersection and union of the predicted box and the true label. The larger the value, the closer the predicted box size is to the true target size. For example, mAP0.5 and mAP0.75 represent the detection accuracy when the IoU threshold is 0.5 and 0.75 respectively. The experimental results in this section use mAP0.5 as the standard for judging the detection accuracy of the model.
[0128] 2. Ablation experiment
[0129] In order to evaluate the contribution of the SCER module and SSPD-FPN module to the improvement of network detection accuracy, a stacking method is used to explore the impact of the improvements on the accuracy of the YOLOv5 model. First, the SCER module is added to the Backbone part, and the results are shown in Table 3-1. As can be seen from the table, after the SPD module is added to the YOLOv5 benchmark model, the detection accuracy is improved by 1.5%, and the accuracy of small target pests is also improved to a certain extent, indicating that the SPD module retains the detailed features of small targets well during the backbone network transmission process, effectively alleviating the disappearance of small target features.
[0130] Table 3-1 Comparison of SCER moduleablation experimental results
[0131]
[0132] In order to further explore the impact of the SPD module on the feature extraction of the YOLOv5 network, the SPD module is cumulatively deployed behind the four convolutional layers in Backone in sequence. After the ablation experiment, the results are shown in Table 3-2:
[0133] Table 3-2 Comparison of SPD module ablation experiments
[0134]
[0135] Although adding SPD modules to all four layers of Backbone has the highest increase in efficiency, compared with adding three layers, the number of parameters increases significantly, and the accuracy improvement is limited. Therefore, it is believed that the reason is that the fourth layer is already a deeper network, and the characteristics of small targets in this layer have basically disappeared. After weighing the pros and cons, the most cost-effective three-layer SPD module was finally selected. Then, based on the SPD module, the C3 module was improved, the ECA attention mechanism was added, and the RFEM module was added at the end of the backbone network. After experiments, the effect was improved by 1.2% on the basis of SPD, which is enough to verify that the ECA attention mechanism helps the C3 module pay more attention to the important features of small targets in the process of feature extraction, and the features have richer multi-scale information after the void convolution with different void rates, which is conducive to the expression of the semantic information features of small targets.
[0136] The above are improvements on the Backbone feature extraction part of the model, while the Neck feature fusion part is mainly divided into two parts: ssFPN and ODConv.
[0137] Table 3-3 Comparison of Neck partialablation experiments
[0138]
[0139] The ablation experiment after adding SSOD-FPN is shown in Table 3-3. Based on the SCER module, the target scale features are obtained by performing 3D convolution on the features of each layer of the feature pyramid, and combined with the high-resolution features. Figure 1 This method effectively alleviates the semantic gap between feature maps and is very helpful for extracting semantic information of small targets. The detection accuracy is also improved by 0.8%, of which the detection accuracy of small targets is improved by about 1%. Then, full-dimensional dynamic convolution is added to the position where the low-resolution feature map and the medium-resolution feature map are fused in the feature pyramid, so that the four dimensions of the convolution kernel can be adaptively adjusted according to the input features, thereby further enhancing the feature extraction capability of small targets.
[0140] Overall, the improvement of the YOLOv5 model has improved its detection accuracy on the mulberry pest dataset by nearly 4%, and small target pests by 2.4%. The 4% improvement includes not only small-sized pests, but also some pest samples that have small target characteristics due to the shooting distance. Not only that, Param is one of the standards for measuring model complexity. The YOLOv5 baseline model has 7127235 parameters, and the improved model has 8871735 parameters, which only increases by 24%. At the same time, the inference time increased from the initial 23.7ms to 27.8ms, which only increased by 17.2%, but the performance has been improved to a certain extent.
[0141] 3. Comparative experiment
[0142] In order to evaluate the advantages and disadvantages of the improved model among many target detection models, some mainstream targets were selected.
[0143] Detection algorithms: Faster-RCNN, RetinaNet, etc., are trained using the same self-built mulberry pest dataset, as shown in Table 3-4.
[0144] Table 3-4 Comparative experimental results
[0145]
[0146] For the above network, the training was uniformly set to 200 Epochs and BatchSize value was 16. The training results are shown in Table 3-4, where YOLOv5-Pest represents the improved model. As can be seen from the table, the model has achieved a high mAP accuracy for both the overall and small target pest samples, which is sufficient to illustrate the effectiveness of the improvement work.
[0147] 4. Visual analysis of experimental results
[0148] In order to explain the improvement of the improved YOLOv5 model proposed in this chapter in the detection of small target pests, the process of network learning small target features is represented in the form of feature heat map, such as Fig.10As shown in the figure, the four pictures in the upper part represent the process of learning features of the baseline YOLOv5 network, and the lower part represents the learning process of the improved network. Among them, a represents the 0th layer of the network, that is, the picture has just been input into the network. It can be seen that the feature responses of multiple small targets in the picture are still obvious. b represents the feature map of the picture after passing through the backbone network. It can be seen that some small targets have lost their responses, but the improved network has less feature loss in comparison. c is the high-resolution feature map output after the features are fused through the feature pyramid. At this time, the YOLOv5 baseline network has lost most of the small target features, while the improved network retains them more completely. d is the final prediction map. It can be seen that the improved YOLOv5 model has a much lower missed detection rate for small pest targets than the baseline network, which is enough to illustrate the effectiveness of the network improvement.
[0149] To solve the problem of low accuracy of mulberry disease detection under complex background, the YOLOv5 model (disease feature extraction module) is improved. Fig.11 shown.
[0150] The work mainly focuses on two parts: feature extraction and feature fusion:
[0151] (1) Drawing on the idea of Res2net multi-scale fusion network, the C3 module of Backbone feature extraction and Neck feature fusion is improved to have a multi-scale receptive field.
[0152] (2) Use the ContextAggregation module at the locations of the three Head detection heads to improve the feature fusion method:
[0153] 1. Feature extraction
[0154] Since the improvement work is based on the YOLOv5 network, and the way the network extracts features is to obtain feature maps of different sizes and channels through the backbone network Darknet-53, and then input them into the FPN+PAN structure. In this process, the C3 module is the main module for learning the residual features of the target, which is mainly distributed after the four convolutional layers of Backbone feature extraction and before the three Concat splicing layers of Neck feature fusion. In order to better adapt to the interference of complex backgrounds on foreground targets, the network needs to learn richer features. Inspired by the Res2net multi-scale fusion network, the feature learning module C3 module is improved so that it can have a multi-scale receptive field, thereby further mining the important features of the target.
[0155] 2. Feature Fusion
[0156] YOLOv5 often fuses feature maps of different sizes through top-down and lateral propagation, thereby fusing shallow graphic features with deep semantic features, and then inputs them into three heads of different sizes for prediction. In order to further enhance the feature representation of the target, the ContextAggregation module is introduced before the prediction head, with the aim of learning the spatial global context information at each level, so as to better enable the network to learn the distinction and connection between the foreground target and the background, and reduce background interference.
[0157] Further detailed description of the improvements:
[0158] 1. Data Augmentation
[0159] During the imaging process, diseases will always interfere with detection due to changes in lighting, scale, shape and background. To solve this problem, we mainly start from two aspects: data source and detection network. The most intuitive way is to increase the diversity of data. Therefore, in terms of data processing, data enhancement is used to simulate different background conditions. There are many ways to enhance data, such as single sample data enhancement, multi-sample data enhancement, etc. Fig.12 shown.
[0160] After screening, 16 data enhancement methods were finally selected, including flipping, scaling, Gaussian noise, grayscale change, etc., and they were made into a random data enhancer. This enhancer combines the enhancement method with the random value. By inputting the image file path to be enhanced and the desired enhancement amount, the data can be randomly enhanced in various ways. Fig.13 As shown, we can see that a picture can contain multiple enhancement methods, which effectively increases the diversity of the data and also improves the robustness and generalization of the model, making it better adaptable to various environmental conditions.
[0161] After data enhancement, the number of mulberry disease datasets has also expanded to 12,238, as shown in Table 4-2:
[0162] Table 4-2 Mulberry disease dataset
[0163]
[0164]
[0165] 2. Model Improvement
[0166] For the problem of mulberry disease detection under complex environmental interference, it is not enough to enhance the data. We should also focus on the detection model YOLOv5. The model is improved mainly from two parts, namely feature extraction and feature fusion. Two modules, C3-Res2block and ContextAggregation, are designed.
[0167] (1) Feature extraction: C3-Res2block multi-scale fusion module
[0168] In complex environments, background interference is often a major factor leading to low target detection accuracy. Noisy or similar environments will cause the network's attention to deviate from the target. Of course, this is related to the fact that the foreground target feature expression is not strong enough. When studying how to further explore the detailed features of the target, it was found that multi-scale representation features are very important, because even if the background at a certain scale interferes with the target, it may be relatively small at other scales. Therefore, background information at different scales can provide the model with rich information about the target's environment, helping the model to understand and detect the target more accurately. However, most convolutional neural networks now express multi-scale features in a hierarchical manner, which will cause some details or local information to be lost during the transmission process. In order to solve this problem, the idea of Res2net is borrowed, that is, the resolution features obtained through different receptive fields are used to improve the feature expression ability at a finer level. The specific implementation method is to replace the 3*3 filter of c channels with a group of smaller filters, and then group the input features. The filter first learns information from the first group of feature maps, and then inputs the output features together with the next group of feature maps into the group of filters again. This is repeated until all group features are processed, and then all feature maps are concat-connected, and then input into the 1*1 filter for fusion, and finally input into the SE module to adaptively adjust the channel, such as Fig.14 shown.
[0169] The filter groups are connected in a residual manner, so that when passing through different filter groups, feature information with different receptive fields will be generated, thereby helping the neural network to extract global and local multi-scale information of the target and improve the target feature expression ability. The improvement work is based on Res2net, and the C3 module of YOLOv5 is designed. C3-Res2block is proposed to make it also have a multi-scale receptive field, thereby improving the deep mining ability of features.
[0170] (2) Feature fusion: Context Aggregation module
[0171] The next improvement is made in the model feature fusion stage. That is, a ContextAggregation module is added before the three Head detection heads of different sizes. This module is inspired by the self-attention mechanism and can fuse the spatial context at the pixel level:
[0172]
[0173] Here P i and Q i They represent the input and output feature maps of the i-th layer in the feature pyramid, and each feature map contains N i pixels, j,m∈{1,N i} represents the index of each pixel, w k and w v is the linear projection matrix used to project the feature map. Formula 4-1 simplifies the self-attention mechanism by replacing the matrix multiplication of query and key with linear projection. In addition, a i and Q i The updated weight matrix of the same shape is used to balance the degree to which each pixel aggregates the spatial context. This matrix can also be simply derived from P i Perform linear projection and generate through Sigmoid activation.
[0174]
[0175] Similarly, j∈{1,N i} is the matrix index, and the output of the Sigmoid function is regarded as the proportion of information aggregated from the global context. The structure diagram of the ContextAggregation module is shown in Figure 15.
[0176] The model uses this module to learn the spatial context information at each level and fuses it into each pixel through adaptive weights. This soft weighting strategy ensures that only useful global information can be fused into local areas without reducing the differences between objects. This improves the expressiveness of target features and reduces the interference caused by background.
[0177] To further illustrate the effect of this embodiment, the following experiment is designed:
[0178] 1. Experimental settings and evaluation indicators
[0179] (1) Experimental environment
[0180] In order to verify the detection effect of the improved mulberry disease detection model under background interference, an ablation experiment and a comparative experiment were designed. The ablation experiment used the mainstream deep learning framework PyTorch, and the comparative experiment used the MMDetection framework. The hardware conditions for model training are shown in Table 4-3:
[0181] Table 4-3 Hardware environment forablation and comparison experiments
[0182]
[0183] The data set required for model training uses a self-built mulberry disease data set. The data samples in this data set are all taken in real environments, so there are a lot of complex backgrounds caused by changes in various environmental conditions such as light and scale.
[0184] (2) Evaluation indicators
[0185] The evaluation indicators of target detection can be divided into two categories, namely model detection effect indicators and model parameter indicators. The model detection effect can also be divided into two categories, one is the accuracy of identifying and classifying target diseases, and the other is the accuracy of locating the size and position of the target. The experimental results use mAP as an indicator to measure the detection accuracy of the target detection model, and use IOU intersection-over-union ratio to measure the degree of overlap between the predicted box and the actual target size and position, such as Fig.16 shown.
[0186] The calculation formula of IoU is:
[0187]
[0188] Among them, C represents the predicted box and G represents the real box. In addition, the number of model parameters and the amount of calculation are also key factors in measuring the quality of the model. Parameters and GFLOPS are often used to represent them in target detection. FLOPS represents the number of floating-point operations per second of the model, which is used to measure the computing power of the processor. The smaller the value, the simpler the model is and the smaller the amount of calculation required.
[0189] 2. Ablation experiment
[0190] In order to evaluate the effectiveness of the improved model, a comparison is made from three aspects and an ablation experiment is designed. From the perspective of the data source, the degree of improvement in the detection accuracy of the model before and after the use of data enhancement is compared; in terms of feature fusion, on the basis of using data enhancement, the effect of adding the ContextAggregation module on the model is compared; in terms of feature extraction, on the basis of previous improvements, the impact of transforming the C3 module into the C3-Res2block on the model is compared. The ablation experiment divides the data set into 1:8:1, the number of experimental rounds is uniformly set to 200Epochs, and the number of samples captured for one training, BatchSize, is set to 16. In order to further compare the effect of the improved model, a training method without pre-trained weights is used during training. The results of the ablation experiment are shown in Table 4-4:
[0191] Table 4-4 Ablation experiment of mulberry disease model
[0192]
[0193] As can be seen from the table, after enhancing the mulberry disease data, the detection effect of the model has been greatly improved, which shows that the enhanced data can simulate various complex background conditions well, so that the limited data set can also allow the model to learn richer knowledge. On the basis of data enhancement, the ContextAggregation spatial context aggregation module is added to the model feature fusion part, which enables the model to further learn the difference between the target feature and the background through context information, thereby improving the model by 0.4 percentage points. Next, the C3 module of the feature extraction part of YOLOv5 is improved to have a multi-scale receptive field, which improves the detection accuracy of the model by 1.8%. Not only that, it can be seen in the above table that after adding the designed C3-Res2block, the amount of calculation has been reduced by 3.8% with only a 2.3% increase in the number of parameters. This is enough to show that this module can not only improve the detection accuracy, but also reduce the unnecessary amount of calculation of the model, thereby reducing the computational burden.
[0194] The above are indicators for the accuracy of discriminating target categories. Fig.17 The table shows the accuracy of the model when the IoU ratio is within 50 and between 50 and 95. As can be seen from the table, the improvement of the disease detection model has corresponding improvements under different IoU ratios, indicating that the improvement work has a comprehensive improvement on the model.
[0195] 3. Comparative experiment
[0196] The mainstream target detection model and the improved mulberry disease model were used for comparative experiments, including the one-stage detection model RetinaNet and the two-stage detection model Faster RCNN. These models were trained using the self-built mulberry disease dataset, and the number of Epochs for experimental training was uniformly set to 200, the number of training samples per batch was set to 16, and no pre-training was used. The comparative experimental results are shown in Figure 2. Fig.18 As shown in the figure, the improved YOLOv5 model outperforms the other two target detection algorithms on the mulberry disease dataset, with an mAP of 88.7%. The main reason is that data enhancement improves the diversity of data, and the combination of ContextAggregation and C3-Res2block modules further improves the network's feature extraction and feature fusion capabilities.
[0197] Further exploration of the detection effect of the model on various diseases, as shown in Table 4-5, shows that the mainstream target detection models have different focuses on disease training. For example, Faster-RCNN has a good detection effect on mulberry powdery mildew, indicating that the model is good at learning large-size disease targets, while RetinaNet is more accurate in detecting mulberry leaf blight, indicating that it is more inclined to learn targets with simple backgrounds. The improved YOLOv5 model can achieve the best detection effect regardless of the type of disease samples, indicating that the proposed improvement method can play a role regardless of the complex background or scale scaling, which is sufficient to verify the effectiveness of the improvement.
[0198] Table 4-5 Comparative experiments on various mulberry disease models
[0199]
[0200] 4. Visual analysis of experimental results
[0201] The previous content has verified the effectiveness of the improvement work from the perspective of ablation experiments and comparative experiments. In order to more intuitively demonstrate the detection effect of the model, the feature heat maps of different stages are compared for illustration, such as Fig.19 shown.
[0202] As can be seen from the picture, the symptom of mulberry leaf blight is that the leaves turn yellow. If the photo is taken under sufficient light, the sunlight will shine through the leaves to form light spots on the ground. The color of the light spots is very similar to that of leaf blight, which can easily lead to misidentification. Fig.19The first row shows the detection of diseases using the YOLOv5 basic model. The detection results show that a mulberry leaf blight was not detected, but the light spot in the background was judged as mulberry leaf blight, which shows the interference of complex background on model detection. The second row of the picture shows the detection effect of the improved model. From the results, it can be seen that the diseased leaves of the two mulberry leaf blight were detected and were not interfered by the background light spots with similar colors, shapes and symptoms. From the feature heat maps of the two models in the feature extraction stage and the fusion stage, it can be seen that the improved model has always focused on the characteristics of the target, while the basic model lost the learning of some disease targets in the feature fusion stage, which is enough to verify the effectiveness of the improvement work on the complex background interference problem in this paper.
[0203] The above are only embodiments of the present invention. The invention is not limited to the fields involved in this implementation case. The common sense such as the known specific structures and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or the priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to explain the content of the claims.
Claims
1. A mulberry tree pest and disease intelligent detection and identification system, characterized in that: include: Video acquisition module, used to collect image data of mulberry tree diseases and pests in real time; An image processing module, used for preprocessing the collected image data; The pest feature extraction module is used to extract the features of pests from the preprocessed images; A disease feature extraction module is used to extract disease features from the preprocessed image; Detection model module, used to identify pests and diseases based on the extracted features; The data management module is used to store and manage the test results.
2. A mulberry tree disease and insect pest intelligent detection and identification system according to claim 1, characterized in that: The pest feature extraction module includes a Backbone unit, a Neck unit and a Head unit; the Backbone unit is used to extract image features, including a SCER module, and the SCER module includes a Focus+SPD submodule, a C3-ECA submodule and a RFEM submodule; the Neck unit is used to fuse features, including an ssFPN unit and an ODConv unit, and the ssFPN unit is used to treat each layer of feature maps in the feature pyramid as each frame in the video for Concat splicing, and then input into the 3D convolution; the ODConv unit is used to introduce full-dimensional dynamic convolution and learn from four dimensions through a multi-dimensional attention mechanism; the Head unit is used to detect and identify pest types.
3. A mulberry tree disease and insect pest intelligent detection and identification system according to claim 2, characterized in that: The Focus+SPD submodule is used to replace the convolutional layer of the 0th layer of the backbone network with the Focus structure, and add an SPD module after each convolutional layer.
4. A mulberry tree disease and insect pest intelligent detection and identification system according to claim 3, characterized in that: The C3-ECA submodule adds the ECA attention mechanism to the C3 module to learn the channel attention of each convolutional block.
5. A mulberry tree disease and insect pest intelligent detection and identification system and method according to claim 4, characterized in that: The RFEM submodule is added after the SPPF spatial pyramid pooling layer, and three dilated convolutions with different dilation rates are used to extract multi-scale information.
6. The intelligent detection and identification system for mulberry pests and diseases according to claim 5, characterized in that: The ssFPN unit adjusts the feature maps of each layer of the feature pyramid to a uniform size through operations such as convolution upsampling, and uses 3D convolution to extract scale sequence features.
7. The intelligent detection and identification system for mulberry pests and diseases according to claim 6, characterized in that: The ODConv unit is added after the two C3 modules in the top-down path, and a multi-dimensional attention mechanism is introduced. The disease feature extraction module includes a C3-Res2block submodule and a Context Aggregation submodule; the Context Aggregation submodule includes an SE unit; the C3-Res2block submodule is used to replace the 3*3 filters of c channels with a group of smaller filters, and then group the input features. The filter first learns information from the first group of feature maps, and then inputs the output features together with the next group of feature maps into the group of filters again, and repeats until all group features are processed, then all feature maps are concat-connected, and then input into the 1*1 filter for fusion, and finally input into the SE unit to adaptively adjust the channel; wherein the filter group is connected in a residual manner; the Context Aggregation submodule is used to fuse the spatial context at the pixel level inspired by the self-attention mechanism: Here P i and Q i They represent the input and output feature maps of the i-th layer in the feature pyramid, and each feature map contains N i pixels, j,m∈{1,N i } represents the index of each pixel, w k and w v It is the linear projection matrix used to project the feature map. Formula 1 replaces the matrix multiplication of query and key by linear projection. Apply a P i and Q i The updated weight matrix of the same shape is used to balance the degree to which each pixel aggregates the spatial context. This matrix can also be simply derived from P i Perform linear projection and generate through Sigmoid activation; Again, j∈{1,N_i} is the matrix index and the output of the Sigmoid function is regarded as the proportion of information aggregated from the global context.
8. A method for intelligent detection and identification of mulberry tree diseases and insect pests, comprising the following steps: Step 1: Collect image data of mulberry tree diseases and insect pests in real time through the video acquisition module; Step 2: Preprocess the collected image data through an image processing module, including but not limited to denoising, contrast enhancement, etc.; Step 3: Extract the features of pests and diseases in the preprocessed image through the feature extraction module; Step 4: Identify the extracted features through the detection model module to determine the type and location of pests and diseases; Step 5: Store and manage the test results through the data management module.
