Pear leaf disease and insect pest segmentation method based on deep learning

Abstract

The present invention discloses a method for segmenting pests and diseases of pear leaves based on deep learning, comprising the following steps: step 1, generating a pest and disease segmentation model, the pest and disease segmentation model is constructed based on a U-Net network and introduces a spatial channel attention module CSA and a channel fusion feature extraction module CFFE, and the pest and disease segmentation model includes an image channel adjustment part, an encoding part, a decoding part, and a fusion feature extraction part; step 2, training the pest and disease segmentation model generated in step 1; step 3, inputting a plant leaf image to be detected into the pest and disease segmentation model trained in step 2, and detecting and segmenting the pest and disease image from the plant leaf image to be detected by the trained pest and disease segmentation model. The present invention enhances the feature extraction and expression capabilities of the model, and also improves the efficiency and robustness of the model, and can adapt to a variety of image processing tasks.

Description

Technical Field

[0001] The present invention relates to the field of plant leaf pest and disease image detection methods, and specifically to a pear leaf pest and disease segmentation method based on deep learning. Background Art

[0002] Image processing and analysis plays a vital role in the field of disease segmentation. In recent years, plant diseases can lead to yield losses. However, manual detection of plant diseases by agricultural experts and botanists is time-consuming and tedious. In order to reduce the risk of disease severity, computer vision technology can play an important role. The severity of the disease can be reduced through computer technology and human collaboration.

[0003] Traditional image algorithms are mainly based on mathematics and computer vision theory, extracting image features such as edges and textures for processing. However, these traditional methods need to be improved in terms of accuracy, robustness and efficiency for the complex environment of plant leaves, especially pear tree leaves. In particular, their performance is often greatly reduced when encountering problems such as occlusion, lighting changes and background.

[0004] In comparison, deep learning, as one of the latest image processing and data analysis technologies, has shown good results and great potential. Currently, deep learning has been widely used in the segmentation and identification of leaf diseases and pests. It can automatically extract effective features from images, greatly improving the ability to identify image diseases. The main advantage of deep learning is that it can automatically learn features in data, extract image features and learn, thereby greatly improving the efficiency and accuracy of data processing. However, the diseases of pear tree leaves are relatively dense and the disease characteristics are not obvious. Deep learning methods still face challenges in the accuracy and efficiency of identification and segmentation.

[0005] In summary, the existing technologies have the following deficiencies and disadvantages when used for plant leaf disease and insect pest detection, especially for pear tree leaf disease and insect pest detection:

[0006] 1. Traditional image processing methods rely on manually designed feature extractors, such as color histograms and texture features. These methods often fail to extract effective features when faced with complex or non-uniformly distributed disease areas, resulting in reduced recognition accuracy.

[0007] 2. Due to the diversity of plant leaf diseases and light variations in the natural environment, these manually designed features are poorly adaptable to diseased areas and are easily disturbed by external factors.

[0008] 3. The existing neural network algorithms are highly complex and have low robustness, making it difficult to meet the needs of disease segmentation.

[0009] 4. The existing network will make misjudgments when processing multiple diseases at the same time, and the recognition results are not stable enough.

[0010] Therefore, a method is now needed that can accurately and efficiently identify diseases and segment specific locations in the above environment in order to control the diseases. The present invention is to solve the above problems and provide a pear leaf disease and insect pest segmentation method based on deep learning. Summary of the invention

[0011] The present invention provides a pear leaf disease and insect pest segmentation method based on deep learning. By proposing a spatial channel attention module CSA and a channel fusion feature extraction module CFFE based on the U-Net architecture, the pear leaf disease images can be effectively segmented and processed, and the accuracy and efficiency of segmentation can be improved to solve the problems existing in the prior art.

[0012] In order to achieve the above object, the technical solution adopted by the present invention is:

[0013] The pear leaf pest and disease segmentation method based on deep learning includes the following steps:

[0014] Step 1: Generate a pest and disease segmentation model. The pest and disease segmentation model is constructed based on the U-Net network and introduces the spatial channel attention module CSA and the channel fusion feature extraction module CFFE. The pest and disease segmentation model includes an image channel adjustment part, an encoding part, a decoding part, and a fusion feature extraction part, wherein:

[0015] The image channel adjustment part obtains the original input image, improves the dimension of the original input image through the image channel adjustment part, and obtains the feature map F after the dimension is adjusted;

[0016] The encoding part includes four pooling layers and four spatial channel attention modules CSA; the feature map F obtained by the image channel adjustment part is first pooled by the first pooling layer and then feature extracted by the first spatial channel attention module CSA to obtain the feature map ; Then, the feature map obtained by the first spatial channel attention module CSA After pooling by the second pooling layer, the second spatial channel attention module CSA performs feature extraction to obtain the feature map ; Then, the feature map obtained by the second spatial channel attention module CSA After pooling by the third pooling layer, the third spatial channel attention module CSA performs feature extraction to obtain the feature map ; Finally, the feature map obtained by the third spatial channel attention module CSA After pooling by the fourth pooling layer, the fourth spatial channel attention module CSA performs feature extraction to obtain the feature map ;

[0017] The decoding part includes four upsampling layers and four skip connection convolution fusion modules; the feature map obtained by the fourth spatial channel attention module CSA in the encoding part First, upsample through the first upsampling layer to obtain the upsampled feature map , and then the upsampled feature map obtained by the first upsampling layer is fused through the first skip connection convolutional fusion module And the feature map obtained by the fourth spatial channel attention module CSA After splicing, convolution is performed to obtain the feature map ; Next, the feature map obtained by the first skip connection convolution fusion module The upsampled feature map is obtained after upsampling by the second upsampling layer , and then the upsampled feature map obtained by the second upsampling layer is fused through the second skip connection convolutional fusion module And the feature map obtained by the third spatial channel attention module CSA After splicing, convolution is performed to obtain the feature map ; Then, the feature map obtained by the second skip connection convolution fusion module The upsampled feature map is obtained after upsampling by the third upsampling layer , and then the upsampled feature map obtained by the third upsampling layer is fused through the third skip connection convolutional fusion module And the feature map obtained by the second spatial channel attention module CSA After splicing, convolution is performed to obtain the feature map ; Finally, the feature map obtained by the third skip connection convolution fusion module The upsampled feature map is obtained after upsampling by the fourth upsampling layer , and then the up-sampled feature map obtained by the fourth up-sampling layer is fused through the fourth skip connection convolutional fusion module And the feature map obtained by the first spatial channel attention module CSA After splicing, convolution is performed to obtain the feature map ;

[0018] The fusion feature extraction part includes a channel fusion feature extraction module CFFE, a feature fusion module FFS, and an output convolution; the channel fusion feature extraction module CFFE obtains the original input image and performs a CFFE operation on the original input image to obtain a feature map ; The feature fusion module FFS obtains the feature map obtained by the fourth skip connection convolution fusion module in the decoding part , the feature map obtained by the channel fusion feature extraction module CFFE , and the feature fusion module FFS combines the feature map and feature map Convolution and batch normalization are performed respectively to obtain the corresponding feature maps , Then the feature fusion module FFS combines the two feature maps , Concatenate on the channel dimension to get the feature map Then the feature map Perform convolution, normalization, After activation, the feature matrix is ​​obtained, and then the feature fusion module FFS converts the feature map , Multiply them by the feature matrix respectively and combine them with the feature map obtained by the channel fusion feature extraction module CFFE And the feature map obtained by the fourth skip connection convolution fusion module Add them separately and get the corresponding result feature map , Finally, the feature fusion module FFS combines the two result feature maps , Add together to get the final result feature map ;

[0019] The output convolution in the fusion feature extraction part is used to obtain the feature map of the feature fusion module FFS. Perform convolution processing to obtain the output result of the pest and disease segmentation model;

[0020] Step 2: training the pest and disease segmentation model generated in step 1;

[0021] Step 3: Input the image of the plant leaves to be detected into the pest and disease segmentation model trained in step 2, and use the trained pest and disease segmentation model to detect and segment the pest and disease image from the image of the plant leaves to be detected.

[0022] Furthermore, in the pest and disease segmentation model generated in step 1, the image channel adjustment part performs convolution, normalization, and ReLU activation on the original input image in sequence to improve the dimension of the original input image.

[0023] Furthermore, in the encoding part of the pest segmentation model described in step 1, each spatial channel attention module CSA first extracts a global feature representation based on the pooled feature map output by the corresponding pooling layer. , local feature representation ; Then based on the local feature representation Get the feature matrix and represent the global features After multiplying with the feature matrix and performing residual connection, the enhanced feature map is obtained ; Then, the enhanced feature map Perform maximum pooling and average pooling respectively, and then enhance the feature map After the maximum pooling and average pooling results are concatenated, convolution and sigmoid activation are performed in sequence to obtain the feature matrix, and then the enhanced feature map is Multiply it with the feature matrix and perform residual connection to obtain the feature map ; Finally, the feature map After convolution, normalization and ReLU activation, the feature map output by the CSA module of each spatial channel is obtained.

[0024] Furthermore, in the fusion feature extraction part described in step 1, the CFFE operation process of the channel fusion feature extraction module CFFE is as follows:

[0025] First, two different convolutions are used to convert the original input image into two different sets of high-dimensional feature representations, and the corresponding feature matrix is ​​obtained based on each set of high-dimensional feature representations; then, the two sets of high-dimensional feature representations are cross-multiplied with the two sets of feature matrices and the results are multiplied to obtain a deep fusion feature map. Next, the feature map After convolution and sigmoid activation, it is multiplied with the original input image to obtain the feature map ; Finally, the feature map After adding to the original input image, a learnable element-by-element scaling operation is performed to obtain the feature map output by the channel fusion feature extraction module CFFE.

[0026] Compared with the prior art, the present invention has significant advantages, which are specifically embodied in the following aspects:

[0027] 1. Multi-level feature extraction and fusion mechanism: This invention realizes deep feature extraction and dynamic fusion between channels by introducing the channel fusion feature extraction module CFFE into the U-Net structure. The channel fusion feature extraction module CFFE uses multi-scale convolution (3*3 and 5*5 convolution kernels) combined with the attention mechanism to adaptively weight features of different sizes, thereby enhancing the capture of key features. This mechanism significantly improves the model's ability to process multi-scale information, and is particularly suitable for fine feature extraction in complex scenes.

[0028] 2. Synergy of channel and spatial attention: By introducing the spatial channel attention module CSA, the present invention enhances the model's attention to important features while maintaining the network's high efficiency. The spatial channel attention module CSA combines the two mechanisms of channel attention and spatial attention, and performs weighted operations on the channel and spatial dimensions respectively, thereby achieving more accurate feature enhancement. This design enables the model to adaptively adjust the feature strength when processing complex areas in the image to ensure the effective expression of important areas.

[0029] 3. Innovative design of feature map feature fusion: By performing element-by-element weighted adjustment on the feature map in the upsampling path and the feature map after feature extraction, efficient information fusion is achieved. The weight relationship between features is dynamically adjusted to optimize the effect of information transmission and enhance the expressiveness of the network in fine-grained feature expression.

[0030] 4. Learnable element-by-element scaling operation: In the feature extraction process, the present invention combines the element-by-element scaling operation and introduces learnable scaling parameters to perform element-by-element adaptive adjustment of the feature map. This operation further enhances the dynamic expression ability of the features and ensures that the feature strength of each channel is reasonably optimized, thereby improving the flexibility and robustness of the model.

[0031] 5. Efficiency and scalability of the structure: This invention cleverly combines multi-scale feature extraction, channel and spatial attention mechanism, and innovative design of jump connection, so that the entire network can perform fine-grained processing of features of different scales and complexities while ensuring high efficiency. At the same time, the model achieves significant performance improvement while ensuring a low number of parameters, and has good scalability and application potential.

[0032] This paper significantly improves the performance of the U-Net model in image segmentation tasks by optimizing structural features and innovating functional modules, especially when processing complex images and multi-scale features. These improvements not only enhance the model's feature extraction and expression capabilities, but also improve the model's efficiency and robustness, making it adaptable to a variety of image processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a diagram of the pest and disease segmentation model architecture of an embodiment of the present invention.

[0034] Figure 2 It is a diagram of the CSA architecture of the spatial channel attention module according to an embodiment of the present invention.

[0035] Figure 3 It is an architecture diagram of the channel fusion feature extraction module CFFE according to an embodiment of the present invention.

[0036] Figure 4It is a diagram of the FFS architecture of the feature fusion module of an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0038] This embodiment discloses a method for segmenting pear leaf pests and diseases based on deep learning, comprising the following steps:

[0039] Step 1: Generate a pest and disease segmentation model. The pest and disease segmentation model generated in this embodiment is constructed based on a U-Net network.

[0040] The main network of the existing U-Net framework is an encoder-decoder architecture. First, the original input image is transformed in dimension to facilitate the subsequent input of features into the network. Then there are 4 downsampling layers, which mainly extract features from the image and transfer the features to the corresponding upsampling part through jump connections. Then there are 4 upsampling layers, which are used to combine the corresponding transferred features for feature fusion and restore the image size and dimension.

[0041] like Figure 1 As shown, the pest segmentation model of this embodiment is constructed by introducing the spatial channel attention module CSA and the channel fusion feature extraction module CFFE based on the prior art U-Net framework, including the image channel adjustment part, the encoding part, the decoding part, and the fusion feature extraction part. The data processing process of the pest segmentation model of this embodiment is as follows:

[0042] (1) Image channel adjustment part

[0043] The image channel adjustment part obtains the original input image In the image channel adjustment part, the original input image is Perform the first convolution to improve the original input image The dimension of the feature map is adjusted , in order to facilitate subsequent feature input, the process of image channel adjustment is as follows:

[0044]

[0045] in: is the original input image; ReLU is the activation function, and BN is the batch normalization operation; It is a convolution operation with a convolution size of 3*3; is the output feature map, with a size of , where C, H, and W are the number of channels, height, and width of the feature image R, respectively.

[0046] (2) Coding part

[0047] The encoding part includes four pooling layers and four spatial channel attention modules CSA. A total of four pooling layers and four spatial channel attention modules CSA feature extraction are performed, as follows:

[0048] 2.1) The first pooling and the first spatial channel attention module CSA for feature extraction

[0049] The feature map F obtained by the image channel adjustment part is first pooled by the first pooling layer with a pooling kernel size of 2*2 and a step size of 2 to obtain the feature map , the process is shown as follows:

[0050]

[0051] in: is the maximum pooling operation; the output feature map size is .

[0052] The first spatial channel attention module CSA obtains the feature map of the first pooling layer For feature extraction, the first spatial channel attention module CSA can efficiently extract important features of the feature map and greatly reduce the number of parameters and calculations of the original method. The process is shown in the following formula:

[0053]

[0054] in: It is the feature map obtained and output by the first spatial channel attention module CSA, and its size is .

[0055] 2.2) Second pooling and second spatial channel attention module CSA for feature extraction

[0056] The second pooling layer obtains the feature map of the first spatial channel attention module CSA Perform a pooling operation with a pooling kernel size of 2*2 and a step size of 2 to obtain the feature map , the process is shown as follows:

[0057]

[0058] in: For the maximum pooling operation, the output feature map size is .

[0059] The second spatial channel attention module CSA obtains the feature map of the second pooling layer The feature extraction process is as follows:

[0060]

[0061] in: The feature map obtained and output by the second spatial channel attention module CSA has a size of .

[0062] 2.3) The third pooling and the third spatial channel attention module CSA for feature extraction

[0063] The third pooling layer obtains the feature map of the second spatial channel attention module CSA Perform a pooling operation with a pooling kernel size of 2*2 and a step size of 2 to obtain the feature map , the process is shown as follows:

[0064]

[0065] in: For the maximum pooling operation, the output feature map size is .

[0066] The third spatial channel attention module CSA obtains the feature map of the third pooling layer The feature extraction process is as follows:

[0067]

[0068] in: It is the feature map obtained and output by the third spatial channel attention module CSA, and its size is .

[0069] 2.4) The fourth pooling and the fourth spatial channel attention module CSA for feature extraction

[0070] The fourth pooling layer obtains the feature map of the third spatial channel attention module CSA Perform a pooling operation with a pooling kernel size of 2*2 and a step size of 2 to obtain the feature map , the process is shown as follows:

[0071]

[0072] in: For the maximum pooling operation, the output feature map size is .

[0073] The fourth spatial channel attention module CSA obtains the feature map of the fourth pooling layer The feature extraction process is as follows:

[0074]

[0075] in: This is the feature map obtained and output by the fourth spatial channel attention module CSA, and its size is .

[0076] like Figure 2 As shown, in this embodiment, each spatial channel attention module CSA combines channel attention and spatial attention mechanisms, aiming to achieve focus on key information through multi-scale feature extraction and pixel-by-pixel enhancement. And fully integrate features at different levels to improve the network's ability to identify important features. The data processing process of each spatial channel attention module CSA is as follows:

[0077] A1) First, each spatial channel attention module CSA performs multi-scale feature extraction on the pooled feature map output by the corresponding pooling layer through 3*3 and 1*1 convolution operations, normalization and ReLU activation, thereby extracting the global feature representation , local feature representation , the process is shown as follows:

[0078]

[0079] in: Indicates that convolution is performed with a convolution kernel of size 3*3; Indicates convolution with a convolution kernel of size 1*1; BN indicates batch normalization; ReLU is the activation function; Represents the pooled feature map of the corresponding pooling layer output (i.e. or or or ).

[0080] A2) Next, the local feature representation After the maximum pooling operation and normalization, the sigmoid activation function is used to obtain the feature matrix, and the global feature representation is After multiplying with the feature matrix, the residual connection is used to realize the adaptive weighted adjustment of the features, and the expression ability of important channels is enhanced, thereby obtaining the enhanced feature map , the process is shown as follows:

[0081]

[0082] in: represents the sigmoid activation function, which effectively captures the key information; BN represents the batch normalization operation; It is the maximum pooling operation.

[0083] A3) Enhanced feature map Perform maximum pooling and average pooling respectively to obtain the maximum pooling , average pooling results , the process is as follows:

[0084]

[0085] in: represents the average pooling operation;

[0086] Then the enhanced feature map And its maximum pooling result , average pooling results After splicing in the channel dimension, convolution with a convolution kernel size of 1*1 is used in turn to change the number of channels to 1, and the feature matrix is ​​obtained through the sigmoid activation function. Then, the enhanced feature map Multiply it with the feature matrix and perform residual connection to enhance the spatial information, thus obtaining the feature map , the process is shown as follows:

[0087]

[0088] Finally, the feature map Through 3*3 convolution processing, normalization and ReLU activation, the feature map output by the attention module CSA of each spatial channel is obtained. (Right now or or or ), the process is shown as follows:

[0089]

[0090] Through the synergistic effect of channel and spatial attention, the spatial channel attention module CSA can significantly enhance the network's ability to capture and express multi-scale information, especially showing strong feature recognition in complex scenes.

[0091] (3) Decoding part

[0092] The decoding part includes four upsampling layers and four skip connection convolution fusion modules. A total of four upsampling and four skip connection convolution fusions are performed, as follows:

[0093] 3.1) First upsampling and skip connection convolution fusion

[0094] First, the feature map obtained by the fourth spatial channel attention module CSA in the encoding part is obtained through the first upsampling layer Upsample to get the upsampled feature map The process is as follows:

[0095]

[0096] Among them: the first upsampling layer uses transposed convolution to upsample, and the upsampling feature map .

[0097] Then, the upsampled feature map obtained by the first upsampling layer is fused through the first skip connection convolutional fusion module. And the feature map obtained by the fourth spatial channel attention module CSA Perform the splicing operation, the process is as follows:

[0098]

[0099] in: For the splicing operation, and The feature map obtained by splicing in the channel dimension .

[0100] Then, the feature map obtained by the concatenation operation is fused through the first skip connection convolution module Perform a single convolution operation, normalization and ReLU activation to obtain the feature map , the process is as follows:

[0101]

[0102] in: is a convolution with a kernel size of 3*3, BN is a batch normalization operation, ReLU is the activation function, and the feature map .

[0103] 3.2) Second upsampling and skip connection convolution fusion

[0104] The feature map obtained by the first skip connection convolution fusion module through the second upsampling layer Upsample to get the upsampled feature map The process is as follows:

[0105]

[0106] Among them: the second upsampling layer uses transposed convolution to upsample, and the feature map .

[0107] The upsampled feature map obtained by the second upsampling layer is then fused through the second skip connection convolutional fusion module. And the feature map obtained by the third spatial channel attention module CSA Perform the splicing operation, the process is as follows:

[0108]

[0109] in: For the splicing operation, and The feature map obtained by splicing in the channel dimension .

[0110] Then, the feature map obtained by the splicing operation is fused through the second skip connection convolution module Perform a single convolution operation, normalization and ReLU activation to obtain the feature map , the process is as follows:

[0111]

[0112] in: is a convolution with a kernel size of 3*3, BN is a batch normalization operation, ReLU is the activation function, and the feature map .

[0113] 3.3) The third upsampling and skip connection convolution fusion

[0114] The feature map obtained by the second skip connection convolution fusion module through the third upsampling layer Upsample to get the upsampled feature map The process is as follows:

[0115]

[0116] Among them: the third upsampling layer uses transposed convolution to upsample, and the feature map .

[0117] The upsampled feature map obtained by the third upsampling layer is then fused through the third skip connection convolutional fusion module. And the feature map obtained by the second spatial channel attention module CSA Perform the splicing operation, the process is as follows:

[0118]

[0119] in: For the splicing operation, and The feature map obtained by splicing in the channel dimension .

[0120] Then, the feature map obtained by the splicing operation is fused through the third skip connection convolution module Perform a single convolution operation, normalization and ReLU activation to obtain the feature map , the process is as follows:

[0121]

[0122] in: is a convolution with a kernel size of 3*3, BN is a batch normalization operation, ReLU is the activation function, and the feature map .

[0123] 3.4) The fourth upsampling and skip connection convolution fusion

[0124] The feature map obtained by the fourth upsampling layer of the third skip connection convolution fusion module Upsample to get the upsampled feature map The process is as follows:

[0125]

[0126] Among them: the fourth upsampling layer uses transposed convolution to upsample, and the feature map .

[0127] The upsampled feature map obtained by the fourth upsampling layer is then fused through the fourth skip connection convolutional fusion module. And the feature map obtained by the first spatial channel attention module CSA Perform the splicing operation, the process is as follows:

[0128]

[0129] in: For the splicing operation, and The feature map obtained by splicing in the channel dimension .

[0130] Then, the feature map obtained by the splicing operation is fused through the fourth skip connection convolution module Perform a single convolution operation, normalization and ReLU activation to obtain the feature map , the process is shown as follows:

[0131]

[0132] in: is a convolution with a kernel size of 3*3, BN is a batch normalization operation, ReLU is the activation function, and the feature map .

[0133] (4) Fusion feature extraction part

[0134] The fusion feature extraction part includes the channel fusion feature extraction module CFFE, the feature fusion module FFS, and the output convolution. The data processing process is as follows:

[0135] 4.1) Channel Fusion Feature Extraction Module CFFE

[0136] The original input image is obtained by the channel fusion feature extraction module CFFE , and for the original input image Perform CFFE operation to obtain the characteristic map , the process is shown as follows:

[0137]

[0138] in: Indicates that the channel fusion feature extraction module CFFE performs CFFE operation, For the original input image, get the feature map .

[0139] 4.2) Feature Fusion Module FFS

[0140] like Figure 4 As shown, first, the feature fusion module FFS obtains the feature map obtained by the fourth skip connection convolution fusion module in the decoding part , the feature map obtained by the channel fusion feature extraction module CFFE , the feature fusion module FFS combines the feature map and feature map Convolution is performed separately, and batch normalization is performed to obtain the corresponding feature map , The process is as follows:

[0141]

[0142] Where: BN represents batch normalization operation; It means using a 1*1 convolution kernel for convolution, changing the number of channels of the feature map to 1 to facilitate subsequent operations.

[0143] Then, the feature fusion module FFS combines the two feature maps , The process of splicing in the channel dimension is as follows:

[0144]

[0145] in: For splicing operation.

[0146] The feature fusion module FFS then fusions the feature map Perform convolution and normalization, and then use The activation function is converted into a feature matrix as shown below:

[0147]

[0148] in: represents the sigmoid activation function, BN represents the batch normalization operation, Indicates convolution with a 1*1 convolution kernel.

[0149] Then, the feature fusion module FFS combines the feature maps , Multiply them by the feature matrix respectively and combine them with the feature map obtained by the channel fusion feature extraction module CFFE And the feature map obtained by the fourth skip connection convolution fusion module Add the corresponding results to get the feature map , , the process is shown as follows:

[0150]

[0151] Finally, the feature fusion module FFS combines the two result feature maps , Add together to get the final result feature map , the process is shown as follows:

[0152]

[0153] The feature fusion module FFS is constructed by and Element-by-element weighted adjustment is performed to achieve efficient information fusion. Compared with the simple splicing operation in traditional U-Net, this module uses multi-scale convolution and Sigmoid activation function to dynamically adjust the weight relationship between features, thereby optimizing the effect of information transmission and improving the network's expressiveness in fine-grained feature expression.

[0154] 4.3) Output Convolution

[0155] Output convolution feature fusion module FFS to obtain the feature map Perform 3*3 convolution to obtain the output result of the pest and disease segmentation model , the process is shown as follows:

[0156]

[0157] like Figure 3 As shown, in this embodiment, the channel fusion feature extraction module CFFE implements a feedforward neural network with a channel aggregation mechanism, which gradually extracts and fuses feature information between channels, combines attention mechanism and scaling operation to enhance the feature expression ability of the network. This module can not only realize fine-grained fusion of features in multi-scale convolution operations, but also dynamically adjust the importance of features through attention mechanism. The data processing process of the channel fusion feature extraction module CFFE is as follows:

[0158] B1) First, two different convolutions are used to transform the original input image Converted into two different sets of high-dimensional feature representations, and obtained the corresponding feature matrix based on each set of high-dimensional feature representations.

[0159] Specifically, one of the convolutions is a convolution with a kernel size of 3*3, which transforms the original input image into The number of channels from Expand to ,in Indicates the number of channels of the original input image, so that the input image is converted into a higher-dimensional feature representation, and the high-dimensional feature representation obtained by the 3*3 convolution Perform sigmoid activation function to obtain feature matrix , the process is shown as follows:

[0160]

[0161] in: represents the sigmoid activation function, Indicates that convolution is performed with a convolution kernel of size 3*3. Indicates that the 3*3 convolution result is a high-dimensional feature representation. Represents the corresponding feature matrix.

[0162] Another convolution is a convolution with a kernel size of 5*5. The original input image is convolved by a 5*5 convolution. The number of channels from Expand to , the input image is converted into a higher-dimensional feature representation, and the high-dimensional feature representation obtained by the 5*5 convolution is Perform sigmoid activation function to obtain feature matrix , the process is shown as follows:

[0163]

[0164] in: represents the sigmoid activation function, Indicates that convolution is performed with a convolution kernel of size 5*5. Indicates that the 5*5 convolution result is a high-dimensional feature representation. Represents the corresponding feature matrix. This multi-scale convolution operation provides rich contextual information for subsequent feature fusion.

[0165] B2) Representing two sets of high-dimensional features , With two sets of feature matrices , After cross multiplication, multiply the results again to get a deep fusion feature map This interactive attention mechanism combines two feature representations of different scales in the channel dimension, further strengthening the network's attention to important features. The process is shown in the following formula:

[0166]

[0167] Through the interactive attention mechanism, the convolution results of two different scales are cross-multiplied, and the interactive results are further multiplied, ultimately achieving deep fusion of features and improving the network's ability to model inter-channel correlation.

[0168] B3) Deep fusion feature map Perform a 3*3 convolution to increase the number of channels from Zoom to Then, after sigmoid activation, the feature matrix is ​​obtained and multiplied by the original input image. , get the feature map , the process is shown as follows:

[0169]

[0170] in: represents the sigmoid activation function, Indicates that convolution is performed with a convolution kernel of size 3*3. is the original input image.

[0171] B4) Finally, the feature map With the original input image After addition, a learnable element-by-element scaling operation is performed to obtain the feature map output by the channel fusion feature extraction module CFFE (i.e., feature map ).

[0172] Specifically, the original input image With feature map Add, then introduce a learnable element-by-element scaling operation to adjust the merged feature map. A learnable parameter is defined in the scaling operation and the gradient is updated during the back propagation process. The parameter scales each channel in the feature map in an element-by-element manner and ensures that the learnable element-by-element scaling operation is only performed on the channel dimension, retaining spatial information, thereby improving the expressiveness of the model. The process is shown in the following formula:

[0173]

[0174] in: represents a learnable element-wise scaling operation.

[0175] The learnable element-by-element scaling operation ensures fine-grained adjustment of feature expression and is optimized through back-propagation, effectively improving the network's adaptability and robustness to complex tasks.

[0176] In this embodiment, the design of the spatial channel attention module CSA and the channel fusion feature extraction module CFFE fully reflects the efficient extraction and fusion of multi-scale features. Through the synergy of channel and spatial dimensions and the learnable adaptive scaling mechanism, these two modules significantly enhance the feature expression ability of the network and provide strong support for improving the performance of the model.

[0177] Step 2: training the pest and disease segmentation model generated in step 1;

[0178] The pear leaf disease image data used in this embodiment mainly comes from Dangshan County, Suzhou City, Anhui Province, Yingshang County, Fuyang City, and Tianmen Town, Tongling City. In the orchard environment, there are uncontrollable natural factors such as leaf overlap, occlusion, water droplets, and changes in light at different angles. In order to fully capture the different postures of the leaves, we used a variety of shooting methods to record images from multiple angles. Due to the small size of the data set, in order to further enrich the data, this embodiment selected images from the public plant disease image data set DiaMOS Plant Dataset (Fenu_Malloci_2021) for expansion. The image selection process is strictly controlled to ensure the effectiveness of the experiment, providing a solid and extensive data foundation for subsequent research.

[0179] In order to facilitate the deep learning network to learn the data set and meet the needs of related segmentation tasks, this embodiment combines data enhancement technology to further enrich the data set. Specifically, this embodiment uses a variety of data enhancement methods, including: (a) image rotation, by rotating the image at any angle to improve the spatial distribution of the data; (b) image brightness adjustment, by randomly changing the saturation, contrast, brightness and sharpness of the image to simulate the characteristics of diseases in different environments; (c) random enhancement, the image is randomly enhanced according to a certain probability to enhance the model's adaptability in complex situations.

[0180] The experiment was implemented using Pytorch and run on an NVIDIA 4090 GPU. The pear leaf dataset is divided into a training set and a test set in a ratio of 7:3, and the two parts have no intersection.

[0181] In the process of training the data set, this embodiment uses 100 epochs for training. This is based on the preliminary experimental results of this embodiment, which determine that a balance can be achieved between efficiency and accuracy in 100 cycles. In addition, this embodiment sets the batchsize to 8, and the learning rate is set to be dynamically adjusted between 0.0001 and 0.000001. This embodiment selects Adam as the optimizer with a momentum of 0.9. Adam combines the two optimization strategies of RMSProp and Momentum, and can dynamically adjust the learning rate to reduce gradient oscillation, while accelerating the model convergence speed through the momentum mechanism.

[0182] During the optimization process, this embodiment adopts a combination strategy of cross entropy loss and Dice loss. This combination can not only improve the model's ability to identify and locate pear leaf spots in complex environments, but also effectively deal with the problem of class imbalance. Dice loss reduces the model's bias in dealing with minority categories by directly optimizing the overlap between the predicted area and the true area.

[0183]

[0184] Where: t is the true label, y is the predicted output of the model, s is a hyperparameter to ensure the stability of the calculation and its value is set to 1e-5, L 1 is the cross entropy loss, which is used to measure the difference between the model prediction and the true label, L 2 Dice loss is used to optimize the overlap between the predicted area and the true area. and are weight coefficients, set to 1 and 0.5 respectively, and Loss is the total loss.

[0185] Step 3: Input the image of the plant leaves to be detected into the pest and disease segmentation model trained in step 2, and use the trained pest and disease segmentation model to detect and segment the pest and disease image from the image of the plant leaves to be detected.

[0186] In the improvement of the U-Net model of this embodiment, by introducing new technical means, the problems of insufficient multi-scale feature extraction and feature fusion in the existing pear leaf disease image segmentation network are solved, and the synergy of channel attention and spatial attention is further optimized, which has the following significant advantages:

[0187] 1. Channel fusion feature extraction module CFFE: This embodiment introduces the channel fusion feature extraction module CFFE into the U-Net structure to achieve deep feature extraction and dynamic fusion between channels. The channel fusion feature extraction module CFFE uses multi-scale convolution (3*3 and 5*5 convolution kernels) combined with the attention mechanism to adaptively weight features of different sizes, thereby enhancing the capture of key features. This mechanism significantly improves the model's ability to process multi-scale information, and is particularly suitable for fine feature extraction in complex scenes.

[0188] 2. Spatial channel attention module CSA: By introducing the spatial channel attention module CSA, this embodiment enhances the model's attention to important features while maintaining network efficiency. The spatial channel attention module CSA combines the two mechanisms of channel attention and spatial attention, and performs weighted operations on the channel and spatial dimensions respectively, thereby achieving more accurate feature enhancement. This design enables the model to adaptively adjust feature strength when processing complex areas in the image to ensure effective expression of important areas.

[0189] 3. Innovative design of feature map feature fusion: By performing element-by-element weighted adjustment on the feature map in the upsampling path and the feature map after feature extraction, efficient information fusion is achieved. The weight relationship between features is dynamically adjusted to optimize the effect of information transmission and enhance the expressiveness of the network in fine-grained feature expression.

[0190] 4. Learnable element-by-element scaling operation: In the feature extraction process, this embodiment combines the element-by-element scaling operation and introduces learnable scaling parameters to perform element-by-element adaptive adjustment of the feature map. This operation further enhances the dynamic expression ability of the feature and ensures that the feature strength of each channel is reasonably optimized, thereby improving the flexibility and robustness of the model.

[0191] It can be seen that this embodiment significantly improves the performance of the U-Net model in the pear leaf disease image segmentation task through the optimization of structural features and the innovation of functional modules, especially when processing complex pear leaf disease image images and multi-scale features.

[0192] The pest segmentation model of this embodiment is described below through experiments. The experiments were conducted under the Pytorch framework. All experiments were conducted in the same environment. The specific hardware of the experiments was: GPU: NVIDIA GeForce RTX 4090, CPU: el® Core™ i9-13900K Processor, 32G memory, 64-bit Windows 11 operating system, and CUDA was used for training.

[0193] Compared with other segmentation networks, this experiment compares the following indicators:

[0194] MIoU (Mean Intersection over Union): MIoU is a commonly used evaluation indicator in image segmentation tasks, which represents the ratio of the intersection and union of the predicted result and the true label. The calculation method is: the average of the intersection over union (IoU) of each category. The higher the MIoU value, the more accurate the segmentation result.

[0195] MPA (Mean Pixel Accuracy): MPA represents the pixel accuracy of each category. It calculates the ratio of correctly predicted pixels of each category to the total pixels, and then averages the accuracy of all categories. MPA focuses on the accuracy of overall pixel classification, but is less sensitive to small objects. The higher the MPA, the better the network performs in overall pixel classification.

[0196] Dice Similarity Coefficient (DSC): The Dice coefficient mainly measures the ratio of the overlapping part of two sets to the total number. The closer the value is to 1, the greater the overlap and the better the segmentation effect.

[0197] Parameters: Parameters refer to the number of parameters of the model, reflecting the complexity and computational complexity of the model. Generally, fewer parameters mean a lighter model with lower computational requirements; however, too few parameters may affect the expressiveness of the model. The comparative experimental data is shown in Table 1:

[0198] Table 1 Experimental comparison data table

[0199] Model MIoU(%) MPA(%) Dice(%) Parameters(M) Unet 79.73 88.52 88.18 7.08 SegNet 79.49 87.16 88.07 29.44 FCN 84.63 90.18 91.44 32.94 DeepLabv3+ 81.85 91.60 89.67 39.63 PspNet 75.88 85.70 85.58 49.06 HRNet 76.53 86.84 85.89 29.53 Unet++ 83.26 89.26 90.57 26.90 Transunet 77.43 85.33 86.58 63.72 Pest segmentation model in this embodiment 86.5 91.85 92.58 22.68

[0200] As can be seen from Table 1, the pest and disease segmentation model of this embodiment achieves 86.50% in the MIoU index, which shows that its segmentation effect has been significantly improved. Compared with the comparative segmentation methods of other experiments, the pest and disease segmentation model of this embodiment has better segmentation performance in almost all other indicators. MIoU is a key indicator of the segmentation task. The superior performance of the model of this embodiment is mainly attributed to the fact that the introduced attention mechanism can effectively capture the required features and ignore some useless information. At the same time, the introduced feature extraction module can combine multi-scale convolutions and perform channel aggregation to effectively enhance the detail features. The M in Parameters (M) in Table 1 is Million, which is used to represent the unit of the number of parameters of the model.

[0201] The excellent performance of the pest and disease segmentation model of this embodiment in the mean pixel accuracy (MPA) and Dice coefficient experiments shows that the model can effectively locate and accurately identify the boundary area of ​​pear leaf diseases. In comparison, the segmentation effect of other models in this data set is relatively inferior. In contrast, the MPA and Dice values ​​of the pest and disease segmentation model of this embodiment reached 91.85% and 92.58% respectively, showing a significant improvement. The above experimental results fully verify the superior performance of the pest and disease segmentation model of this embodiment in the plant disease segmentation task. And the parameter amount of the pest and disease segmentation model of this embodiment is 22.68M, which is within an acceptable range and is much smaller than the parameter amount of models such as Deeplabv3+ and SwinUnet.

[0202] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction, and such combinations should also be regarded as the contents disclosed in the present disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0203] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by technical personnel in this field should fall within the protection scope of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A pear leaf pest and disease segmentation method based on deep learning, characterized in that: The following steps are involved: Step 1: Generate a pest and disease segmentation model. The pest and disease segmentation model is constructed based on the U-Net network and introduces the spatial channel attention module CSA and the channel fusion feature extraction module CFFE. The pest and disease segmentation model includes an image channel adjustment part, an encoding part, a decoding part, and a fusion feature extraction part, wherein: The encoding part includes four pooling layers and four spatial channel attention modules CSA; The decoding part includes four upsampling layers and four skip connection convolution fusion modules; The image channel adjustment part obtains the original input image and obtains the feature map F after dimension adjustment; The encoding part obtains the dimension-adjusted feature map F, and the feature map is extracted by the encoding part ; The decoding part obtains the feature map obtained by the encoding part , the feature map is extracted from the decoding part ; The fusion feature extraction part obtains the original input image and the feature map obtained by the decoding part The fusion feature extraction part includes a channel fusion feature extraction module CFFE, a feature fusion module FFS, and an output convolution; the channel fusion feature extraction module CFFE obtains the original input image, and performs a CFFE operation on the original input image to obtain a feature map ; The feature fusion module FFS obtains the feature map obtained by the fourth skip connection convolution fusion module in the decoding part , the feature map obtained by the channel fusion feature extraction module CFFE , and the feature fusion module FFS combines the feature map and feature map Convolution and batch normalization are performed respectively to obtain the corresponding feature maps , Then the feature fusion module FFS combines the two feature maps , Concatenate on the channel dimension to get the feature map , and then the feature map Perform convolution, normalization, After activation, the feature matrix is ​​obtained, and then the feature fusion module FFS converts the feature map , Multiply them by the feature matrix respectively and combine them with the feature map obtained by the channel fusion feature extraction module CFFE And the feature map obtained by the fourth skip connection convolution fusion module Add them separately and get the corresponding result feature map , ， Finally, the feature fusion module FFS module combines the two result feature maps , Add together to get the final result feature map ; Step 2: training the pest and disease segmentation model generated in step 1; Step 3: Input the image of the plant leaves to be detected into the pest and disease segmentation model trained in step 2, and use the trained pest and disease segmentation model to detect and segment the pest and disease image from the image of the plant leaves to be detected.

2. The pear leaf pest segmentation method based on deep learning according to claim 1, characterized in that: In step 1, the image channel adjustment part obtains the original input image, improves the dimension of the original input image through the image channel adjustment part, and obtains the feature map F after dimension adjustment; The feature map F obtained by the image channel adjustment part is first pooled by the first pooling layer and then extracted by the first spatial channel attention module CSA to obtain the feature map ; Then, the feature map obtained by the first spatial channel attention module CSA After pooling by the second pooling layer, the second spatial channel attention module CSA performs feature extraction to obtain the feature map ; Then, the feature map obtained by the second spatial channel attention module CSA After pooling by the third pooling layer, the third spatial channel attention module CSA performs feature extraction to obtain the feature map ; Finally, the feature map obtained by the third spatial channel attention module CSA After pooling by the fourth pooling layer, the fourth spatial channel attention module CSA performs feature extraction to obtain the feature map ; The feature map obtained by the fourth spatial channel attention module CSA in the encoding part First, upsample through the first upsampling layer to obtain the upsampled feature map , and then the upsampled feature map obtained by the first upsampling layer is fused through the first skip connection convolutional fusion module And the feature map obtained by the fourth spatial channel attention module CSA After splicing, convolution is performed to obtain the feature map ; Next, the feature map obtained by the first skip connection convolution fusion module The upsampled feature map is obtained after upsampling by the second upsampling layer , and then the upsampled feature map obtained by the second upsampling layer is fused through the second skip connection convolutional fusion module And the feature map obtained by the third spatial channel attention module CSA After splicing, convolution is performed to obtain the feature map ; Then, the feature map obtained by the second skip connection convolution fusion module The upsampled feature map is obtained after upsampling by the third upsampling layer , and then the upsampled feature map obtained by the third upsampling layer is fused through the third skip connection convolutional fusion module And the feature map obtained by the second spatial channel attention module CSA After splicing, convolution is performed to obtain the feature map ; Finally, the feature map obtained by the third skip connection convolution fusion module The upsampled feature map is obtained after upsampling by the fourth upsampling layer , and then the up-sampled feature map obtained by the fourth up-sampling layer is fused through the fourth skip connection convolutional fusion module And the feature map obtained by the first spatial channel attention module CSA After splicing, convolution is performed to obtain the feature map ; The output convolution in the fusion feature extraction part is used to obtain the feature map of the feature fusion module FFS. Convolution processing is performed to obtain the output result of the pest and disease segmentation model.

3. The pear leaf pest segmentation method based on deep learning according to claim 1 or 2, characterized in that: In the pest and disease segmentation model generated in step 1, the image channel adjustment part performs convolution, normalization, and ReLU activation on the original input image in sequence to improve the dimension of the original input image.

4. The pear leaf pest segmentation method based on deep learning according to claim 1 or 2, characterized in that: In the encoding part of the pest segmentation model described in step 1, each spatial channel attention module CSA first extracts the global feature representation based on the pooled feature map output by the corresponding pooling layer , local feature representation ; Then based on the local feature representation Get the feature matrix and represent the global features After multiplying with the feature matrix and performing residual connection, the enhanced feature map is obtained ; Then, the enhanced feature map Perform maximum pooling and average pooling respectively, and then enhance the feature map After the maximum pooling and average pooling results are concatenated, convolution and sigmoid activation are performed in sequence to obtain the feature matrix, and then the enhanced feature map is Multiply it with the feature matrix and perform residual connection to obtain the feature map ; Finally, the feature map After convolution, normalization and ReLU activation, the feature map output by the CSA module of each spatial channel is obtained.

5. The pear leaf pest segmentation method based on deep learning according to claim 1 or 2, characterized in that: In the fusion feature extraction part described in step 1, the CFFE operation process of the channel fusion feature extraction module CFFE is as follows: First, two different convolutions are used to convert the original input image into two different sets of high-dimensional feature representations, and the corresponding feature matrix is ​​obtained based on each set of high-dimensional feature representations; then, the two sets of high-dimensional feature representations are cross-multiplied with the two sets of feature matrices and the results are multiplied to obtain a deep fusion feature map. Next, the feature map After convolution and sigmoid activation, it is multiplied with the original input image to obtain the feature map ; Finally, the feature map After adding to the original input image, a learnable element-by-element scaling operation is performed to obtain the feature map output by the channel fusion feature extraction module CFFE.

Images