Crop leaf disease image semantic segmentation method

Abstract

The invention discloses a crop leaf disease image semantic segmentation method. The method comprises the following steps: collecting crop leaf disease images, manually labeling the crop leaf disease images, and establishing a crop leaf disease image data set; performing image enhancement on the labels in the training set; preprocessing the crop leaf disease image data set; adding a multi-scale context aggregation module between an encoder and a decoder of the Unet network to obtain a feature extraction neural network model; constructing a loss function; training the feature extraction neural network model of the migration parameter weight by adopting a training set subjected to online data enhancement to obtain a semantic segmentation network model; and obtaining a to-be-segmented crop leaf disease image, inputting the to-be-segmented crop leaf disease image into the semantic segmentation network model, and outputting a semantic segmentation result by the semantic segmentation network model. The method can quickly and accurately perform semantic segmentation on the crop leaf disease image.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a semantic segmentation method for crop leaf disease images. Background Art

[0002] Crop leaf diseases are one of the major problems in agricultural production around the world. Leaf diseases have a serious impact on the growth, development and yield of crops. Semantic segmentation of crop leaf disease images can help to identify and monitor diseases in a timely and accurate manner, so as to take effective measures to prevent and control diseases and improve crop yield and quality. However, this task faces many challenges, including the diversity of leaf disease morphology, large intra-class differences, and high inter-class similarities, which make it difficult to accurately segment leaf disease areas.

[0003] At present, domestic and foreign scholars' research on the identification of crop leaf disease types is concentrated in the field of image classification, and there are relatively few technologies on semantic segmentation. Similar problems such as the appearance of multiple different types of diseases in a leaf cannot be solved by image classification methods alone, and in practical applications, the recognition accuracy is not high, which brings inconvenience to agricultural production. Compared with the current mainstream image classification methods, the semantic segmentation task allows multiple objects to be identified and segmented in an image at the same time. It can assign each pixel in the image to a specific semantic category, which plays a key role in segmenting the contours and boundaries of objects.

[0004] In the field of semantic segmentation, crop leaf diseases have the characteristics of large intra-class differences, high inter-class similarities, and diversity in the size and location of diseased areas. In existing research, there is no solution to these characteristics, resulting in low accuracy of semantic segmentation. At the same time, the performance of the semantic segmentation model depends largely on the quality of the given label (ground truth), and manually annotated labels often have inaccurate boundary definitions. In existing research, there is no way to enhance the quality of labels, resulting in the model being unable to fully learn features. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a semantic segmentation method for crop leaf disease images, which makes the manually annotated labels closer to the real leaf disease area by performing image enhancement on the labels, and better captures and processes complex leaf disease features by adding a multi-scale context aggregation module between the encoder and decoder of the Unet network, thereby enabling fast and accurate semantic segmentation of crop leaf disease images.

[0006] In order to solve the above problems, the present invention adopts the following technical solutions:

[0007] A method for semantic segmentation of crop leaf disease images of the present invention comprises the following steps:

[0008] S1: Collect crop leaf disease images, manually label the crop leaf disease images, establish a crop leaf disease image dataset, and divide the crop leaf disease image dataset into a training set and a test set;

[0009] S2: Image enhancement of labels in the training set;

[0010] S3: Preprocessing of crop leaf disease image dataset;

[0011] S4: Add a multi-scale context aggregation module between the encoder and decoder of the Unet network, replace the up-sampled transposed convolution of the Unet network with bilinear interpolation, and obtain a feature extraction neural network model;

[0012] S5: Construct a loss function to evaluate the gap between the true value and the predicted value;

[0013] S6: Use the training set after online data enhancement to train the feature extraction neural network model of the migration parameter weights, and use the trained feature extraction neural network model as the semantic segmentation network model;

[0014] S7: Use the test set to test the semantic segmentation network model;

[0015] S8: Obtain the diseased leaf image of the crop to be segmented, input the diseased leaf image of the crop to be segmented into a semantic segmentation network model, and the semantic segmentation network model outputs a semantic segmentation result.

[0016] Preferably, in step S1, the crop leaf disease images are collected, the crop leaf disease images are manually labeled, and the method for establishing the crop leaf disease image dataset is as follows:

[0017] Collect original images of different types of crop leaf diseases, use image annotation tools to manually label the diseased areas on the collected original images, and obtain a crop leaf disease image dataset;

[0018] The labeling method is to draw the diseased area on the original image and assign semantic labels to it.

[0019] Preferably, the method for performing image enhancement on the label in step S2 is as follows:

[0020] Convert the original label image into a grayscale image to obtain the grayscale image matrix I gray ;

[0021] Generate grayscale image matrix I grayA noise matrix of uniform size, where each value in the noise matrix is ​​a random value drawn from a standard normal distribution;

[0022] Multiply the noise matrix by the weight η to obtain the weighted noise matrix;

[0023] The grayscale image matrix I gray Add it to the weighted noise matrix to get the grayscale image matrix I after adding noise noisy ;

[0024] Use edge detection algorithm to detect grayscale image matrix I noisy Perform edge detection to obtain a mask;

[0025] The mask is inverted and applied to the original image of the label to obtain the image-enhanced label.

[0026] Preferably, step S3 comprises the following steps: converting all images in the crop leaf disease image dataset into JPG format, and then compressing all images and adjusting the size of the images to 512×512 pixels.

[0027] Preferably, the feature extraction neural network model includes an encoder, a decoder, and a multi-scale context aggregation module, the encoder includes a conv1 layer, a conv2 layer, a conv3 layer, a conv4 layer, and a conv5 layer arranged in sequence, the decoder includes an up_concat4 layer, an up_concat3 layer, an up_concat2 layer, an up_concat1 layer, and a final layer arranged in sequence, and the multi-scale context aggregation module includes an atrous_conv1 layer, an atrous_conv1 layer, an atrous_convl layer, an atrous_conv1 layer, a global_pooling layer, and a Concat layer.

[0028] Preferably, the atrous_conv1 layer is used to generate a 32×32×512 feature map through a 512-channel convolution with a convolution kernel size of 1×1, and then use a global context module to perform feature fusion and enhancement to output a 32×32×512 feature map;

[0029] The atrous_conv2 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 6, and a channel number of 512, and then use the global context module to perform feature fusion and enhancement to output a 32×32×512 feature map;

[0030] The atrous_conv3 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 12, and a channel number of 512, and then use the global context module to perform feature fusion and enhancement, and output a 32×32×512 feature map;

[0031] The atrous_conv4 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 18, and a channel number of 512, and then use the global context module to perform feature fusion and enhancement, and output a 32×32×512 feature map;

[0032] The global_pooling layer is used to perform global average pooling on the 32×32×512 feature map output by the conv5 layer, reducing the feature map size from 32×32×512 to 1×1×512, and then using a 1×1 convolution kernel to perform 512-channel convolution, and finally using the ReLU activation function to enhance the nonlinear feature expression and output a 1×1×512 global pooling feature map;

[0033] The Concat layer is used to superimpose the output feature maps of the atrous_conv1 layer, atrous_conv2 layer, atrous_conv3 layer, atrous_conv4 layer, and global_pooling layer in the depth dimension to generate a comprehensive feature map of 32×32×2560, and use a convolution with a convolution kernel size of 1×1 to convolve the 32×32×2560 comprehensive feature map to reduce the number of channels to 512. Finally, the ReLU activation function is used to enhance the nonlinear feature expression, and finally a feature map of 32×32×512 is output to the up_concat4 layer.

[0034] Preferably, the up_concat4 layer is used to upsample the 32×32×512 feature map output by the Concat layer through bilinear interpolation to obtain a 64×64×512 feature map, merge the 64×64×512 feature map with the 64×64×512 feature map output by the conv4 layer to obtain a 64×64×1024 feature map, and the 64×64×1024 feature map is convolved twice with a convolution kernel size of 3×3 and a channel number of 512, and then further refined by a ReLU activation function, and finally output a 64×64×512 feature map;

[0035] The up_concat3 layer is used to upsample the 64×64×512 feature map output by the up_concat4 layer through bilinear interpolation to obtain a 128×128×512 feature map, merge the 128×128×512 feature map with the 128×128×256 feature map output by the conv3 layer to obtain a 128×128×768 feature map, and perform two convolutions with a kernel size of 3×3 and a channel number of 256 on the 128×128×768 feature map, and further refine the features through the ReLU activation function, and finally output a 128×128×256 feature map;

[0036] The up_concat2 layer is used to upsample the 128×128×256 feature map output by the up_concat3 layer through bilinear interpolation to obtain a 256×256×256 feature map, merge the 256×256×256 feature map with the 256×256×128 feature map output by the conv2 layer to obtain a 256×256×384 feature map, and perform two convolutions with a kernel size of 3×3 and a channel number of 128 on the 256×256×384 feature map, and further refine the features through a ReLU activation function, and finally output a 256×256×128 feature map;

[0037] The up_concat1 layer is used to upsample the 256×256×128 feature map output by the up_concat2 layer through bilinear interpolation to obtain a 512×512×128 feature map, merge the 512×512×128 feature map with the 512×512×64 feature map output by the conv1 layer to obtain a 512×512×192 feature map, and perform two convolutions with a kernel size of 3×3 and a channel number of 64 on the 512×512×192 feature map, and further refine the features through the ReLU activation function, and finally output a 512×512×64 feature map;

[0038] The final layer is used to perform convolution with a kernel size of 1×1 on the 512×512×64 feature map output by the up_concat1 layer, and processes it using K output channels to map the 64-channel features of each pixel to the predicted probabilities of K categories.

[0039] Preferably, step S5 comprises the following steps:

[0040] Constructing the cross entropy loss function L CE :

[0041]

[0042] Constructing Dice loss function LDice :

[0043]

[0044] Constructing the hybrid loss function L total :

[0045] L total =α*L Dice +β*L CE ,

[0046] Where N is the number of pixels, D is the number of categories, and T i，c is the true label of the i-th pixel in class c, is the predicted probability of the i-th pixel in the c-th class, α and β are weight coefficients;

[0047] The mixed loss function L total As a loss function used to evaluate the gap between the true value and the predicted value.

[0048] Preferably, the method for training the feature extraction neural network model of the migration parameter weights using the online data enhanced training set in step S6 is as follows:

[0049] M1: Download the model parameter weights of the Unet network pre-trained in the public image dataset Imagenet, and perform transfer learning on the feature extraction neural network model;

[0050] M2: The training set is used to train the feature extraction neural network model after transfer learning. The following operations are performed on the training set before each iterative training process: the image samples in the training set are uniformly randomly cropped and resized to images of 512×512 pixels; the image samples are uniformly randomly horizontally flipped with a probability of 0.5; the image samples are uniformly randomly occluded with a probability of 0.2 and an aspect ratio ranging from 2 / 100 to 1 / 3.

[0051] Preferably, the optimization strategy for training the feature extraction neural network model uses the Adam method, the initial learning rate is set to 0.0001, and the number of training epochs is set to 150.

[0052] The beneficial effects of the present invention are as follows: by enhancing the semantic segmentation label image, the manually annotated label is made closer to the real leaf disease area; by adding a multi-scale context aggregation module between the encoder and the decoder of the Unet network, the multi-scale context feature information of the image is obtained, and the complex leaf disease features are better captured and processed; the outline of small disease spots can be better segmented through the constructed hybrid loss function; the purpose of stably segmenting the crop leaf disease image is achieved, the segmentation accuracy is improved, and it has a more efficient recognition effect, a smoother convergence process, and better robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of an embodiment;

[0054] Figure 2 It is a comparison diagram of the label before and after image enhancement;

[0055] Figure 3 It is a structural diagram of the feature extraction neural network model;

[0056] Figure 4 It is the segmentation effect diagram of the semantic segmentation network model.

[0057] In the figure: 1. Encoder, 2. Decoder, 3. Multi-scale context aggregation module. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0059] Embodiment: This embodiment provides a method for semantic segmentation of crop leaf disease images, such as Figure 1 As shown, the following steps are included:

[0060] S1: Collect crop leaf disease images, manually label the crop leaf disease images, establish a crop leaf disease image dataset, and divide the crop leaf disease image dataset into a training set and a test set. The ratio of the training set to the test set is 9:1;

[0061] S2: Image enhancement of labels in the training set;

[0062] S3: Preprocessing of crop leaf disease image dataset;

[0063] S4: Add a multi-scale context aggregation module between the encoder and decoder of the Unet network, replace the up-sampled transposed convolution of the Unet network with bilinear interpolation, and obtain a feature extraction neural network model;

[0064] S5: Construct a loss function to evaluate the gap between the true value and the predicted value;

[0065] S6: Use the training set after online data enhancement to train the feature extraction neural network model of the migration parameter weights, and use the trained feature extraction neural network model as the semantic segmentation network model;

[0066] S7: Use the test set to test the semantic segmentation network model;

[0067] S8: Obtain the diseased leaf image of the crop to be segmented, input the diseased leaf image of the crop to be segmented into a semantic segmentation network model, and the semantic segmentation network model outputs a semantic segmentation result.

[0068] In step S1, crop leaf disease images are collected, and the crop leaf disease images are manually labeled to establish a crop leaf disease image dataset as follows:

[0069] Collect original images of different types of crop leaf diseases, use the image annotation tool Labelme to manually label the diseased areas on the collected original images, and obtain a crop leaf disease image dataset;

[0070] The labeling method is: draw the diseased area on the original image and assign it a semantic label. After the labeling is completed, save the labeling result as a JSON file, and convert the JSON file into a binary label image through code.

[0071] The method for image enhancement of the label in step S2 is as follows:

[0072] Convert the original label image into a grayscale image to obtain the grayscale image matrix I gray ;

[0073]

[0074] Among them, I ij represents the pixel value at position (i, j) of the grayscale image matrix,

[0075] Generate grayscale image matrix I gray A noise matrix N of uniform size, where each value in the noise matrix N is a random value drawn from a standard normal distribution;

[0076]

[0077] Among them, N ij is a random value drawn from a standard normal distribution,

[0078] Multiply the noise matrix N by the weight η to obtain the weighted noise matrix G, G = η*N;

[0079] The grayscale image matrix I grayAdd it to the weighted noise matrix G to get the grayscale image matrix I after adding noise noisy , I noisy =I gray +η*N;

[0080] Use the Canny edge detection algorithm to detect the grayscale image matrix I noisy Perform edge detection to obtain a mask;

[0081] E=Canny(I noisy , T1, T2),

[0082] Among them, E is the binary edge detection result matrix, with a shape of H×W; I noisy is a grayscale image matrix with a shape of H×W; T1 is the low threshold in the Canny edge detection algorithm; T2 is the high threshold in the Canny edge detection algorithm;

[0083] The mask is inverted and applied to the original image of the label to obtain the image-enhanced label.

[0084] By adding noise, uncertainty is introduced near the label contour, causing the contour to change slightly. This scheme generates an array of the same size as the input label by adding a random value generated by a normal distribution to each pixel of the grayscale image. Each value in the array is randomly drawn from a standard normal distribution. η is a coefficient that controls the intensity of the noise. The η value is multiplied by the random noise generated by the normal distribution and then added to the original grayscale image. The coefficient η is allowed to adjust the amount of added noise, thereby controlling the degree of noise impact on the image. Choosing a suitable η value can both enhance the edge of the image contour and maintain the natural quality of the image.

[0085] like Figure 2 As shown in the figure, the Ground Truth image is a partial enlargement of the real label. a, b, c, and d are schematic diagrams of the Ground Truth image after four image enhancements. It can be seen that due to the addition of noise, the effect of each enhancement of the Ground Truth image is different, ensuring randomness. From the Ground Truth image, it can be seen that since manual annotation is difficult to be precise to the pixel level, the contour produces many smooth straight lines, but the actual lesion area will hardly have such neat shapes and lines. This problem has been improved by using the label contour enhancement method.

[0086] Step S3 includes the following steps: converting all images in the crop leaf disease image dataset into JPG format, then compressing and resizing all images to images of 512×512 pixels and converting them into a tensor format that can be input into the model.

[0087] Feature extraction neural network models, such as Figure 3 As shown, it includes an encoder 1, a decoder 2, and a multi-scale context aggregation module 3 (GC_ASPP module). The encoder 1 includes a conv1 layer, a conv2 layer, a conv3 layer, a conv4 layer, and a conv5 layer arranged in sequence. The decoder 2 includes an up_concat4 layer, an up_concat3 layer, an up_concat2 layer, an up_concat1 layer, and a final layer arranged in sequence. The multi-scale context aggregation module 3 includes an atrous_conv1 layer, an atrous_conv1 layer, an atrous_conv1 layer, an atrous_conv1 layer, a global_pooling layer, and a Concat layer.

[0088] The encoder performs 4 times of 2-fold downsampling, and each downsampling reduces the size of the input feature map to half of its original size. The image is input into the network, and the backbone network extracts features from the original image to obtain feature maps of four different scales (1 / 2, 1 / 4, 1 / 8, 1 / 16). Then the deepest features are sent to the multi-scale context aggregation module.

[0089] Define X (l) represents the input feature map of the lth layer, W (l) represents the convolution kernel weight matrix of the lth layer, ReLU is the activation function, MaxPool2d is the 2×2 maximum pooling operation, and the formula for the downsampling part can be expressed as:

[0090]

[0091] in, and are the weight matrices for the first and second convolution operations in layer l, respectively.

[0092] Define the feature map input to the multi-scale context aggregation module as X, and the dimension of the feature map is (N, 512, H, W), where N is the batch size, C is the number of channels, and H and W are the height and width, respectively. When the feature map is input to the multi-scale context aggregation module, C = 512.

[0093] The atrous_conv1 layer is used to generate a 32×32×512 feature map of the 32×32×512 feature map output by the conv5 layer through a 512-channel convolution with a convolution kernel size of 1×1, and then use the global context module to perform feature fusion and enhancement, and output a 32×32×512 feature map. The global context module includes a global average pooling layer and two linear transformation layers.

[0094] The feature map passes through the atrous_conv1 layer:

[0095]

[0096] in, is the weight matrix of the 1×1 convolution, is the weight of the fully connected layer in the global context block, σ is the Sigmoid function, and ⊙ represents element-wise multiplication.

[0097] The atrous_conv2 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 6, and a channel number of 512. The global context module is then used for feature fusion and enhancement to output a 32×32×512 feature map.

[0098] The feature map passes through the atrous_conv2 layer:

[0099]

[0100] in, is the weight matrix of a 3×3 convolution with dilation rate 6, are the fully connected layer weights in the global context block.

[0101] The atrous_conv3 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 12, and a channel number of 512. The global context module is then used for feature fusion and enhancement to output a 32×32×512 feature map.

[0102] The feature map passes through the atrous_conv3 layer:

[0103]

[0104] in, is the weight matrix of a 3×3 convolution with dilation rate 12, are the fully connected layer weights in the global context block.

[0105] The atrous_conv4 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 18, and a channel number of 512. The global context module is then used for feature fusion and enhancement to output a 32×32×512 feature map.

[0106] The feature map passes through the atrous_conv4 layer:

[0107]

[0108] in, is the weight matrix of a 3×3 convolution with dilation rate 18, are the fully connected layer weights in the global context block.

[0109] The global_pooling layer is used to perform global average pooling on the 32×32×512 feature map output by the conv5 layer, reducing the feature map size from 32×32×512 to 1×1×512, and then using a 1×1 convolution kernel to perform 512-channel convolution. Finally, the ReLU activation function is used to enhance the nonlinear feature expression and output a 1×1×512 global pooling feature map.

[0110] The feature map passes through the global_pooling layer:

[0111]

[0112] in, is the weight matrix of the 1×1 convolution.

[0113] The Concat layer is used to superimpose the output feature maps of the atrous_conv1 layer, atrous_conv2 layer, atrous_conv3 layer, atrous_conv4 layer, and global_pooling layer in the depth dimension to generate a comprehensive feature map of 32×32×2560. The 32×32×2560 comprehensive feature map is convolved with a convolution kernel size of 1×1 to reduce the number of channels to 512. Finally, the ReLU activation function is used to enhance the nonlinear feature expression, and finally a feature map of 32×32×512 is output to the up_concat4 layer.

[0114] The atrous_conv1 layer, atrous_conv2 layer, atrous_conv3 layer, atrous_conv4 layer, and global_pooling layer are respectively recorded as the 1st, 2nd, 3rd, 4th, and 5th branches of the multi-scale context aggregation module;

[0115] Concatenate the outputs of these five branches along the channel dimension:

[0116] X concat =Concat(X1,X2,X3,X4,X5),

[0117] At this time, X concat The dimension is (N, 2560, H, W). Finally, it passes through a 1×1 convolutional layer and a ReLU activation function:

[0118]

[0119] in, It is the weight matrix of the 1×1 convolution, which reduces the number of channels from 2560 to 512. The dimension of the final output feature map Y is (N, 512, H, W).

[0120] Assume that the expansion rate of the i-th branch of the multi-scale context aggregation module is d i , then the formula can be expressed as:

[0121]

[0122] The multi-scale context module combines the spatial pyramid pooling and the global context module. It contains four multi-scale context branches and one global pooling branch. Finally, the feature maps of the five branches are merged. In the multi-scale context branch, four different atrous convolution layers capture different scale information through convolution kernels of different sizes and different dilation rates. The global information of the feature map is obtained through global average pooling. The global features are mapped to the required output space through the fully connected layer and nonlinear activation function, and the weights are generated through the Sigmoid activation function for weighted fusion with the original features. The important information in the input features is enhanced and the unimportant information is suppressed, thereby improving the feature representation ability. The ReLU activation function is used to activate the features of the positive part and suppress the features of the negative part, thereby increasing the expressiveness and robustness of the model. Through the splicing operation, the feature information of different scales is integrated into a feature map, thereby enriching the feature representation and enhancing the expressiveness of the multi-scale and global context features.

[0123] The up_concat4 layer is used to upsample the 32×32×512 feature map output by the Concat layer through bilinear interpolation to obtain a 64×64×512 feature map, and merge the 64×64×512 feature map with the 64×64×512 feature map output by the conv4 layer to obtain a 64×64×1024 feature map. The 64×64×1024 feature map is convolved twice with a kernel size of 3×3 and a channel number of 512, and then the features are further refined through the ReLU activation function, and finally a 64×64×512 feature map is output.

[0124] The up_concat3 layer is used to upsample the 64×64×512 feature map output by the up_concat4 layer through bilinear interpolation to obtain a 128×128×512 feature map, and merge the 128×128×512 feature map with the 128×128×256 feature map output by the conv3 layer to obtain a 128×128×768 feature map. The 128×128×768 feature map is convolved twice with a kernel size of 3×3 and a channel number of 256, and then the features are further refined through the ReLU activation function, and finally a 128×128×256 feature map is output.

[0125] The up_concat2 layer is used to upsample the 128×128×256 feature map output by the up_concat3 layer through bilinear interpolation to obtain a 256×256×256 feature map, and merge the 256×256×256 feature map with the 256×256×128 feature map output by the conv2 layer to obtain a 256×256×384 feature map. The 256×256×384 feature map is convolved twice with a kernel size of 3×3 and a channel number of 128, and then the features are further refined through the ReLU activation function, and finally a 256×256×128 feature map is output.

[0126] The up_concat1 layer is used to upsample the 256×256×128 feature map output by the up_concat2 layer through bilinear interpolation to obtain a 512×512×128 feature map, and merge the 512×512×128 feature map with the 512×512×64 feature map output by the conv1 layer to obtain a 512×512×192 feature map. The 512×512×192 feature map is convolved twice with a kernel size of 3×3 and 64 channels, and then the ReLU activation function is used to further refine the features, and finally a 512×512×64 feature map is output.

[0127] The final layer is used to perform a convolution with a kernel size of 1×1 on the 512×512×64 feature map output by the up_concat1 layer, using K output channels for processing, and mapping the 64-channel features of each pixel to the predicted probabilities of K categories.

[0128] After the feature map passes through the multi-scale context aggregation module, it enters the decoder. The decoder gradually restores the spatial resolution of the feature map through upsampling and convolution operations. For each upsampling block, the main operations include upsampling, feature map concatenation, and two convolution operations. is the upsampled feature map of the lth layer, Represents the feature map of the jump connection from the corresponding layer of the encoder, the output feature map Y of the lth layer(l) It can be expressed as:

[0129]

[0130] The unified formula for the entire upsampling process is:

[0131] (1) Upsampling operation:

[0132]

[0133] (2) Feature map concatenation and convolution operation:

[0134]

[0135] in, and It is the weight matrix of the l-th layer convolution operation, Upsample represents the bilinear interpolation upsampling operation, Concat represents the feature map concatenation operation in the channel dimension, and scale factor represents the multiple of the feature map size expansion.

[0136] Step S5 includes the following steps:

[0137] Constructing the cross entropy loss function L CE :

[0138]

[0139] Constructing Dice loss function L Dice :

[0140]

[0141] Constructing the hybrid loss function L total :

[0142] L total =α*L Dice +β*L CE ,

[0143] Where N is the number of pixels, D is the number of categories, and T i，c is the true label of the i-th pixel in class c, is the predicted probability of the i-th pixel in the c-th class, α and β are weight coefficients;

[0144] The mixed loss function L total As a loss function used to evaluate the gap between the true value and the predicted value.

[0145] The method for training the feature extraction neural network model of the migration parameter weights using the online data enhanced training set in step S6 is as follows:

[0146] M1: Download the model parameter weights of the Unet network pre-trained in the public image dataset Imagenet, and perform transfer learning on the feature extraction neural network model;

[0147] M2: The training set is used to train the feature extraction neural network model after transfer learning. Before each iteration (batch) training process, the following operations are performed on the training set: the image samples in the training set are uniformly randomly cropped and resized to images of 512×512 pixels; the image samples are uniformly randomly horizontally flipped with a probability of 0.5; the image samples are uniformly randomly occluded with a probability of 0.2 and an aspect ratio ranging from 2 / 100 to 1 / 3, and the image samples are uniformly converted into a tensor format that can be input into the model.

[0148] The optimization strategy for training the feature extraction neural network model uses the Adam method, with the initial learning rate set to 0.0001 and the number of training epochs set to 150.

[0149] After each epoch, the test set is used to verify and evaluate the current model. After the training is completed, the accuracy, average intersection-over-union ratio and other data of each epoch are obtained. The epoch with the best effect is selected to obtain the feature extraction neural network model as the semantic segmentation network model.

[0150] To train the feature extraction neural network model, the network is built and trained using the Pycharm development environment under the PyTorch deep learning framework.

[0151] The image data in the training set is normalized when input into the feature extraction neural network model, and the image samples are normalized using the mean and standard deviation of the Imagenet dataset.

[0152] The trained feature extraction neural network model is saved as a .pth file, which contains the model structure and parameter weights.

[0153] Step S7 includes the following steps: inputting the crop leaf disease images in the test set into the semantic segmentation network model, the semantic segmentation network model performs prediction and generates segmentation maps, and calculates the mIoU index and mPA index to evaluate the overall performance of the model and the performance on different disease categories. Through these steps, the effectiveness and applicability of the model are verified.

[0154] In this scheme, a semantic segmentation network model for segmenting crop leaf disease images is first trained, and then the semantic segmentation network model is used to quickly segment the crop leaf disease images that need to be segmented. This method enhances the semantic segmentation label image to make the manually annotated label closer to the real leaf disease area; by adding a multi-scale context aggregation module between the encoder and decoder of the Unet network to obtain the multi-scale context feature information of the image, it can better capture and process the complex leaf disease features; through the constructed hybrid loss function, the outline of small disease spots can be better segmented; the purpose of stably segmenting crop leaf disease images is achieved, the segmentation accuracy is improved, and it has a more efficient recognition effect, a smoother convergence process, and good robustness.

[0155] For example, apples and grapes, which are representative of crop leaf diseases, are selected as research objects. Original images of 6 types of leaf diseases of apples and grapes are collected from the public data set PlantVillage, namely apple scab, apple brown spot, apple anthracnose, grape black rot, grape brown spot, and grape ring spot. The original images are screened to ensure that there are 300 images of each type, totaling 1,800 images. A crop leaf disease image data set is established, and a semantic segmentation network model is obtained by training using the method of this embodiment. The images of apple scab leaves, apple brown spot leaves, apple anthracnose leaves, grape black rot leaves, grape brown spot leaves, and grape ring spot leaves to be segmented are obtained and input into the semantic segmentation network model. The semantic segmentation network model outputs the semantic segmentation results, such as Figure 4 shown. Figure 4 The original images in the figure are, from left to right, images of apple black scab leaves, images of apple brown spot leaves, images of apple anthracnose leaves, images of grape black rot leaves, images of grape brown spot leaves, and images of grape ring spot leaves.

Claims

1. A semantic segmentation method for crop leaf disease images, characterized in that: The following steps are involved: S1: Collect crop leaf disease images, manually label the crop leaf disease images, establish a crop leaf disease image dataset, and divide the crop leaf disease image dataset into a training set and a test set; S2: Image enhancement of labels in the training set; S3: Preprocessing of crop leaf disease image dataset; S4: Add a multi-scale context aggregation module between the encoder and decoder of the Unet network, replace the up-sampled transposed convolution of the Unet network with bilinear interpolation, and obtain a feature extraction neural network model; S5: Construct a loss function to evaluate the gap between the true value and the predicted value; S6: Use the training set after online data enhancement to train the feature extraction neural network model of the migration parameter weights, and use the trained feature extraction neural network model as the semantic segmentation network model; S7: Use the test set to test the semantic segmentation network model; S8: Obtain the diseased leaf image of the crop to be segmented, input the diseased leaf image of the crop to be segmented into a semantic segmentation network model, and the semantic segmentation network model outputs a semantic segmentation result.

2. The method for semantic segmentation of crop leaf disease images according to claim 1, characterized in that: In step S1, the method of collecting crop leaf disease images, manually labeling the crop leaf disease images, and establishing a crop leaf disease image dataset is as follows: Collect original images of different types of crop leaf diseases, use image annotation tools to manually label the diseased areas on the collected original images, and obtain a crop leaf disease image dataset; The labeling method is to draw the diseased area on the original image and assign semantic labels to it.

3. The method for semantic segmentation of crop leaf disease images according to claim 1, characterized in that: The method for performing image enhancement on the label in step S2 is as follows: Convert the original label image into a grayscale image to obtain the grayscale image matrix I gray ; Generate grayscale image matrix I gray A noise matrix of uniform size, where each value in the noise matrix is ​​a random value drawn from a standard normal distribution; Multiply the noise matrix by the weight η to obtain the weighted noise matrix; The grayscale image matrix I gray Add it to the weighted noise matrix to get the grayscale image matrix I after adding noise noisy ; Use edge detection algorithm to detect grayscale image matrix I noisy Perform edge detection to obtain a mask; The mask is inverted and applied to the original image of the label to obtain the image-enhanced label.

4. The method for semantic segmentation of crop leaf disease images according to claim 1, characterized in that: The step S3 comprises the following steps: all images in the crop leaf disease image dataset are uniformly converted into JPG format, and then all images are compressed and adjusted to images with a size of 512×512 pixels.

5. The method for semantic segmentation of crop leaf disease images according to claim 1, characterized in that: The feature extraction neural network model includes an encoder, a decoder, and a multi-scale context aggregation module. The encoder includes a conv1 layer, a conv2 layer, a conv3 layer, a conv4 layer, and a conv5 layer arranged in sequence. The decoder includes an up_concat4 layer, an up_concat3 layer, an up_concat2 layer, an up_concat 1 layer, and a final layer arranged in sequence. The multi-scale context aggregation module includes an atrous conv1 layer, an atrous_conv 1 layer, an atrous_conv 1 layer, an atrous_conv 1 layer, a global_pooling layer, and a Concat layer.

6. The method for semantic segmentation of crop leaf disease images according to claim 5, characterized in that: The atrous conv1 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a 512-channel convolution with a convolution kernel size of 1×1, and then use the global context module to perform feature fusion and enhancement to output a 32×32×512 feature map; The atrous conv2 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 6, and a channel number of 512. Then, the global context module is used to perform feature fusion and enhancement, and output a 32×32×512 feature map. The atrous conv3 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 12, and a channel number of 512, and then use the global context module to perform feature fusion and enhancement to output a 32×32×512 feature map; The atrous conv4 layer is used to generate a 32×32×512 feature map of 32×32×512 output by the conv5 layer through a dilated convolution with a convolution kernel size of 3×3, a dilation rate of 18, and a channel number of 512, and then use the global context module to perform feature fusion and enhancement to output a 32×32×512 feature map; The global_pooling layer is used to perform global average pooling on the 32×32×512 feature map output by the conv5 layer, reducing the feature map size from 32×32×512 to 1× 1×512, and then using a 1× 1 convolution kernel to perform 512-channel convolution, and finally using the ReLU activation function to enhance the nonlinear feature expression and output a 1× 1× 512 global pooling feature map; The Concat layer is used to superimpose the output feature maps of the atrous conv1 layer, atrous conv2 layer, atrous_conv3 layer, atrous_conv4 layer, and global_pooling layer in the depth dimension to generate a comprehensive feature map of 32×32×2560, and use a convolution with a convolution kernel size of 1×1 to convolve the 32×32×2560 comprehensive feature map to reduce the number of channels to 512. Finally, the ReLU activation function is used to enhance the nonlinear feature expression, and finally a feature map of 32×32×512 is output to the up_concat4 layer.

7. The method for semantic segmentation of crop leaf disease images according to claim 6, characterized in that: The up_concat4 layer is used to upsample the 32× 32× 512 feature map output by the Concat layer through bilinear interpolation to obtain a 64×64×512 feature map, merge the 64× 64×512 feature map with the 64× 64×512 feature map output by the conv4 layer to obtain a 64× 64×1024 feature map, and the 64× 64×1024 feature map is convolved twice with a convolution kernel size of 3×3 and a channel number of 512, and then further refined by the ReLU activation function, and finally output a 64×64×512 feature map; The up_concat3 layer is used to upsample the 64× 64× 512 feature map output by the up_concat4 layer through bilinear interpolation to obtain a 128× 128× 512 feature map, merge the 128×128×512 feature map with the 128×128×256 feature map output by the conv3 layer to obtain a 128×128×768 feature map, and perform two convolutions with a kernel size of 3×3 and a channel number of 256 on the 128×128×768 feature map, and further refine the features through a ReLU activation function, and finally output a 128×128×256 feature map; The up_concat2 layer is used to upsample the 128×128×256 feature map output by the up_concat3 layer through bilinear interpolation to obtain a 256×256×256 feature map, merge the 256×256×256 feature map with the 256×256×128 feature map output by the conv2 layer to obtain a 256×256×384 feature map, and perform two convolutions with a kernel size of 3×3 and a channel number of 128 on the 256×256×384 feature map, and further refine the features through a ReLU activation function, and finally output a 256×256×128 feature map; The up_concat1 layer is used to upsample the 256×256×128 feature map output by the up_concat2 layer through bilinear interpolation to obtain a 512×512×128 feature map, merge the 512×512×128 feature map with the 512×512×64 feature map output by the conv1 layer to obtain a 512×512×192 feature map, and perform two convolutions with a kernel size of 3×3 and a channel number of 64 on the 512×512×192 feature map, and further refine the features through the ReLU activation function, and finally output a 512×512×64 feature map; The final layer is used to perform convolution with a kernel size of 1×1 on the 512×512×64 feature map output by the up_concat1 layer, and processes it using K output channels to map the 64-channel features of each pixel to the predicted probabilities of K categories.

8. The method for semantic segmentation of crop leaf disease images according to claim 1, characterized in that: The step S5 comprises the following steps: Constructing the cross entropy loss function L CE : Constructing Dice loss function L Dice : Constructing the hybrid loss function L total : L total =α*L Dice +β*L CE , Where N is the number of pixels, D is the number of categories, and T i，c is the true label of the i-th pixel in class c, is the predicted probability of the i-th pixel in the c-th class, α and β are weight coefficients; the mixed loss function L total As a loss function used to evaluate the gap between the true value and the predicted value.

9. The method for semantic segmentation of crop leaf disease images according to claim 1, characterized in that: The method for training the feature extraction neural network model of migration parameter weights using the online data enhanced training set in step S6 is as follows: M1: Download the model parameter weights of the Unet network pre-trained in the public image dataset Imagenet, and perform transfer learning on the feature extraction neural network model; M2: The training set is used to train the feature extraction neural network model after transfer learning. The following operations are performed on the training set before each iterative training process: the image samples in the training set are uniformly randomly cropped and resized to images of 512×512 pixels; the image samples are uniformly randomly horizontally flipped with a probability of 0.5; the image samples are uniformly randomly occluded with a probability of 0.2 and an aspect ratio ranging from 2 / 100 to 1 / 3.

10. The method for semantic segmentation of crop leaf disease images according to claim 9, characterized in that: The optimization strategy for training the feature extraction neural network model uses the Adam method, with the initial learning rate set to 0.0001 and the number of training epochs set to 150.

Images