Tobacco leaf scab segmentation method and system based on multi-scale residual cavity convolution

Abstract

The invention relates to the technical field of image processing, in particular to a tobacco leaf disease spot segmentation method and system based on multi-scale residual cavity convolution, and the method comprises the steps: collecting a plurality of tobacco leaf disease images as original images, and carrying out the data expansion of each collected original image through an image enhancement method, establishing a training set, a verification set and a test set based on the original image and the extended data; a semantic segmentation model is constructed, the semantic segmentation model is formed by stacking two sub-networks, and the two sub-networks are connected through an ROIE + module; in combination with the training set, training the semantic segmentation model by using the constructed loss function; verifying the trained semantic segmentation model by adopting a verification set, and selecting an optimal semantic segmentation model; the test set is adopted to test the optimal semantic segmentation model, and the performance of the optimal semantic segmentation model is evaluated; and inputting a to-be-segmented tobacco leaf disease image into the trained semantic segmentation model to obtain a tobacco leaf disease spot segmentation map. According to the method, the tobacco leaf scab area can be accurately and efficiently segmented.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a tobacco leaf spot segmentation method and system based on multi-scale residual dilated convolution. Background Art

[0002] Tobacco is a key cash crop in my country. my country leads the world in both tobacco planting area and tobacco leaf procurement, accounting for 39.7% of global production. Tobacco is highly susceptible to diseases during its growth process. Failure to monitor and control them can lead to widespread disease, reduced yields, and decreased quality. Therefore, timely and accurate disease diagnosis is crucial for developing preventive measures.

[0003] Currently, existing tobacco leaf lesion segmentation methods are primarily based on traditional image processing and computer vision techniques. For example, Teng Juan et al. first used an improved LoG operator to perform edge detection on tobacco lesion images and then implemented the K-means algorithm to segment brown spot disease lesions. Xu Shuaitao et al. successfully segmented tobacco frogeye and brown spot disease lesions from the background using a saliency detection method based on seed point selection, utilizing target pixels and their features as prior information. However, traditional plant disease segmentation methods are limited by lighting, noise, seed point selection, and complex feature processing, resulting in insufficient accuracy and robustness, and are time-consuming, making them difficult to meet the needs of real-time detection. In recent years, with the rapid development of artificial intelligence, research on plant disease segmentation based on deep learning has made significant progress. Deep learning, with its powerful representation and generalization capabilities, can automatically extract low-level features such as color, texture, and shape, as well as more complex high-level semantic features from plant disease images, achieving accurate segmentation of diseased areas. Yuan et al. proposed an improved DeepLab v3+ network that significantly improved the segmentation performance of grape leaf black rot by introducing a channel attention module and a feature fusion branch. Fu et al. designed an RS-UNet network based on an improved UNet to segment potato leaf lesions, using ResNet50 as the backbone network and introducing the SE attention mechanism to improve segmentation accuracy. Zhao Xiaohu et al. constructed an improved multi-scale disease segmentation algorithm based on U-Net for tomato leaf disease segmentation, with an accuracy of 92.9%. The above studies have shown that plant disease segmentation methods based on deep learning have good accuracy and robustness. However, there are currently few deep learning-based lesion segmentation models specifically designed for tobacco diseases. In addition, due to the large morphological differences of lesions at different positions on the leaves and the blurred and adhered edges, the difficulty and accuracy of lesion segmentation are increased, and the existing technology has many limitations. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems existing in the current tobacco leaf spot segmentation method and propose a tobacco leaf spot segmentation method and system based on multi-scale residual void convolution, which can accurately and efficiently segment the tobacco leaf spot area.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A tobacco leaf spot segmentation method based on multi-scale residual dilated convolution, including:

[0007] Collect multiple tobacco leaf disease images as original images, use image enhancement methods to expand the data of each collected original image, and establish training sets, validation sets, and test sets based on the original images and expanded data;

[0008] Build a semantic segmentation model, which consists of two stacked sub-networks connected by the ROIE+ module;

[0009] Combined with the training set, the constructed loss function is used to train the semantic segmentation model; the trained semantic segmentation model is verified using the validation set to select the optimal semantic segmentation model;

[0010] Use the test set to test the optimal semantic segmentation model and evaluate its performance;

[0011] The tobacco leaf disease image to be segmented is input into the trained semantic segmentation model to obtain the tobacco leaf disease spot segmentation map.

[0012] According to the tobacco leaf spot segmentation method based on multi-scale residual hole convolution of the present invention, further, the first subnetwork and the second subnetwork of the semantic segmentation model are improved from the Unet basic network framework, and the 9 convolution blocks of the Unet network are replaced by 9 MC modules, and a CBAM module is inserted after each MC module.

[0013] According to the tobacco leaf spot segmentation method based on multi-scale residual dilated convolution of the present invention, further, the two sub-networks introduce attention gating in each jump connection, and both use RUFF modules in the upsampling stage. The implementation process of the RUFF module is as follows: ①, the outputs of the last CBAM module in the encoder of the two sub-networks and the first three CBAMs in the decoder are respectively sent to the feature alignment module for processing; ②, the four output feature maps obtained by ① are channel-spliced, and then sequentially processed by 3×3 convolution, BN layer, ReLU activation function, and CBAM module; ③, the output result of ② and the output of the last CBAM module of the sub-network are element-by-element added to obtain the final output feature map of the RUFF module.

[0014] According to the tobacco leaf spot segmentation method based on multi-scale residual hole convolution of the present invention, further, the MC module includes two parts, the first part is a 3×3 convolution, and the second part is composed of 1×1, 3×3, and 5×5 convolutions in parallel, and a residual connection is introduced; the process of the MC module processing the input feature map is as follows: ①, the input feature map is sent to the first part of a 3×3 convolution for convolution operation to obtain the output feature map; ②, the second part of 1×1, 3×3, and 5×5 convolutions are used to further extract features from the output result of ① to obtain three output feature maps of different sizes; ③, the three output results of ② are spliced and fused in the channel dimension; ④, the input feature map in ① is passed through a 1×1 convolution as the residual connection of the MC module; ⑤, the fusion result of ③ and the output result of ④ are added element by element to obtain the final output feature map of the MC module.

[0015] According to the tobacco leaf spot segmentation method based on multi-scale residual hole convolution of the present invention, further, the CBAM module is composed of two parts in series: a channel attention module and a spatial attention module; the process of the CBAM module processing the input feature map is as follows: ①, the input feature map is subjected to global average pooling and global maximum pooling operations respectively to obtain two output feature maps; ②, the two output feature maps of ① are respectively sent to a shared 2-layer perceptron; ③, the two feature maps after ② are added to obtain the channel attention weight; ④, the channel attention weight obtained by ③ is multiplied by the input feature map in ① channel by channel; ⑤, the result of ④ is subjected to global average pooling and global maximum pooling operations respectively to obtain two output feature maps; ⑥, the two output feature maps of ⑤ are channel-spliced, and then the number of channels is reduced through a convolution layer to obtain a spatial attention weight map; ⑦, the spatial attention weight map obtained by ⑥ is multiplied by the feature map obtained by ④ position by position to obtain the final output result of the CBAM module.

[0016] According to the tobacco leaf spot segmentation method based on multi-scale residual dilated convolution of the present invention, an MVCR module is further embedded in the second sub-network; and the output features of the encoder part of the first sub-network are reused. Feature reuse means that in the second sub-network, the output of the encoder MC module and the output of the CBAM module of the corresponding layer in the first sub-network are spliced in the channel dimension and then sent to the CBAM module of the layer of the second sub-network.

[0017] According to the tobacco leaf spot segmentation method based on multi-scale residual hole convolution of the present invention, further, the MVCR module is designed with hole convolution, with a total of 4 branches, which are: 4 3×3 convolutions with expansion rates of 1, 2, 5, and 8 are sequentially connected in series as the first branch, 3 3×3 convolutions with expansion rates of 1, 2, and 5 are sequentially connected in series as the second branch, 2 3×3 convolutions with expansion rates of 1 and 2 are sequentially connected in series as the third branch, and 1 3×3 convolution with expansion rate of 1 is used as the fourth branch; residual connection is introduced in each branch; the process of MVCR module processing input feature map is as follows: ①, the input feature map is sent to the 4 branches for calculation respectively to obtain four output feature maps; ②, the four output feature maps of ① are spliced and fused in the channel dimension to obtain an output feature map; ③ the output feature map of ② is reduced in channel dimension using 1×1 convolution; ④, the output result of ③ is processed by BN layer, ReLU activation function, and Dropout layer in sequence to obtain the final output feature map of the MVCR module.

[0018] According to the tobacco leaf spot segmentation method based on multi-scale residual dilated convolution of the present invention, further, the process of the ROIE+ module processing the input feature map is as follows: ①, the input and output of the first sub-network are dot-multiplied, and then multiplied by the weight α; ②, the result obtained by ① and the result of multiplying the input of the first sub-network by the weight β are added as the final output of the ROIE+ module.

[0019] According to the tobacco leaf lesion segmentation method based on multi-scale residual dilated convolution of the present invention, the loss function is composed of a joint loss function of Dice Loss and Focal Loss, which is defined as follows:

[0020] Loss = αL Dice +βL Focal

[0021] Among them, L Dice is the Dice Loss function, L Focal is the Focal Loss loss function, and α and β are the corresponding weight coefficients.

[0022] Furthermore, the present invention also provides a tobacco leaf spot segmentation system based on multi-scale residual dilated convolution, comprising:

[0023] The dataset construction module is used to collect multiple tobacco leaf disease images as original images, perform data augmentation on each collected original image using image enhancement methods, and establish training sets, validation sets, and test sets based on the original images and augmented data;

[0024] The model building module is used to build a semantic segmentation model. The semantic segmentation model is composed of two stacked sub-networks, which are connected by the ROIE+ module.

[0025] The model training and verification module is used to train the semantic segmentation model using the constructed loss function in combination with the training set; the trained semantic segmentation model is verified using the verification set to select the optimal semantic segmentation model;

[0026] The model evaluation module is used to test the optimal semantic segmentation model using the test set and evaluate the performance of the optimal semantic segmentation model;

[0027] The lesion segmentation module is used to input the tobacco leaf disease image to be segmented into the trained semantic segmentation model to obtain the tobacco leaf lesion segmentation map.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. The present invention proposes a semantic segmentation model based on multi-scale residual dilated convolution, which is mainly used for the accurate segmentation of tobacco leaf lesions. By introducing the MC module, the network is given the ability to autonomously adapt to the size and different shapes of lesions. The MVCR module enables the network to combine different receptive fields to comprehensively capture the local details and global structural information of the image. The RUFF module, a feature fusion method based on the residual paradigm, significantly improves the network's ability to accurately locate edge target pixels and respond to small lesions. In addition, in order to guide the feature extraction network to focus on accurate lesion positioning, the CBAM module and attention gate AG are introduced, which not only eliminates noise interference, but also enables the network to achieve better fusion of shallow features and deep features, local information and global information in the globalization process, greatly improving the integrity of lesion extraction and the clarity of edges, and better solving problems such as large differences in lesion morphology and blurred edges.

[0030] 2. The present invention proposes a method for segmenting tobacco leaf disease images based on a semantic segmentation model based on multi-scale residual dilated convolution. In the tobacco leaf lesion segmentation task, the semantic segmentation model based on multi-scale residual dilated convolution significantly outperforms other mainstream models in lesion CPA, Recall, IoU, F1, and Dice indicators, and is 14.9%, 9.36%, 18.71%, 12.44%, and 8.89% higher than the worst DeepLab v3+, respectively, and 2.3%, 2.14%, 3.34%, 2%, and 1.46% higher than the best Unet, respectively, all of which are significant improvements. At the same time, the model has strong generalization ability and robustness, with a moderate number of parameters and computational complexity, and is suitable for actual tobacco leaf lesion segmentation scenarios, providing a new method and idea for lesion segmentation of tobacco and even other plants. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 2 is a flow chart of a tobacco leaf lesion segmentation method based on multi-scale residual dilated convolution according to an embodiment of the present invention;

[0033] Figure 2 2 is a schematic diagram of the overall structure of a semantic segmentation model based on multi-scale residual dilated convolution according to an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the standard Unet basic network framework structure of an embodiment of the present invention;

[0035] Figure 4 is a schematic structural diagram of the MC module according to an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of the structure of the CBAM module according to an embodiment of the present invention;

[0037] Figure 6 1 is a schematic structural diagram of an AG module according to an embodiment of the present invention;

[0038] Figure 7 Schematic diagram of the structure of the ROIE+ module according to an embodiment of the present invention;

[0039] Figure 8 is a schematic structural diagram of an MVCR module according to an embodiment of the present invention;

[0040] Figure 9 This is an example of a tobacco leaf disease sample image according to an embodiment of the present invention;

[0041] Figure 10 These are the segmentation results of tobacco leaf spots using the Unet network model, PSP network model, DeepLab v3+ network model, FCN network model, SegNet network model, UNET++ network model, DoubleU-Net network model, and semantic segmentation model based on multi-scale residual dilated convolution in the embodiments of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] This embodiment provides a tobacco leaf spot segmentation method based on multi-scale residual hole convolution. Figure 1 As shown, it contains the following contents:

[0044] Step S101: Collect multiple tobacco leaf disease images as original images, perform offline data expansion on each collected original image using image enhancement methods such as flipping, transposition, contrast, brightness, blur, and noise, and outline and annotate the diseased areas in all images. Create a data set based on the original images and the expanded data, and randomly divide it into a training set, a validation set, and a test set in proportion.

[0045] Specifically, the original images were collected from October 14-15, 2023, in Baiyang Town and Gaocun Town, Yiyang County, Luoyang City, Henan Province. They were taken using a Huawei mate30 pro mobile phone. There are 210 original images containing diseases such as powdery mildew, wildfire, and brown spot. The original images were first cropped to 512×512 pixels, and then the data was expanded using Python's Albumentations library and annotated using labelme software. The expanded dataset contains 2,000 original images, which are randomly divided into training, validation, and test sets in a ratio of 8:1:1. The training set contains 1,600 images, the validation set 200 images, and the test set 200 images. Some examples of images in the dataset are as follows: Figure 9 shown.

[0046] Step S102: Construct a semantic segmentation model. The semantic segmentation model is composed of two stacked sub-networks, and the two sub-networks are connected by a module of interest (ROIE+).

[0047] The first and second sub-networks of the semantic segmentation model are both improved from the Unet basic network framework. Each block in the Unet basic network framework (i.e., two cascaded 3×3 convolutions) is replaced with a multi-scale convolutional module (MC). A channel-space attention module (CBAM) is inserted after each MC module, and an attention gate (AG) module is introduced in each skip connection. Both use the feature fusion method based on the residual paradigm (RUFF) module to compensate for detail information during the upsampling phase. In addition, a dense residual dilated convolution module (MVCR) is embedded in the second sub-network, and the output features of the encoder part of the first sub-network are reused.

[0048] like Figure 2 As shown in Figure 2, the construction process of the semantic segmentation model is as follows:

[0049] S21. Build the Unet network.

[0050] This example uses a 30-core AMD EPYC 7742 CPU, an NVIDIA GeForce RTX A6000 GPU, 60.9GB of RAM, CUDA version 11.8, a Linux operating system, Python 3.10.12, and the PyTorch 2.0.1 deep learning framework. The Adam optimizer is used, with an initial learning rate of 0.0001, an input image size of 512×512, and a batch size of 16. Training is performed for 200 epochs.

[0051] The Unet network model consists of two parts: encoder and decoder. Figure 3 As shown in the figure, the encoder is responsible for extracting semantic information from the feature map. It consists of five stacked blocks, gradually expanding the network's receptive field. A max pooling layer is inserted between each block to reduce the spatial resolution of the feature map. The decoder, consisting of four blocks, performs four upsampling operations, gradually restoring the feature map to the original input image size. Finally, a convolutional layer maps the decoder output to pixel-level segmentation results.

[0052] Specifically, the input image size of the Unet network is 512×512×3; the structure of each block is Conv2d+BN+ReLU+Conv2d+BN+ReLU, which is convolution layer, batch normalization operation, activation function, convolution layer, batch normalization operation, activation function, and the convolution kernel size of the convolution layer is 3×3 with a step size of 1; the pooling size used in the maximum pooling layer is 2×2 with a step size of 1; the upsampling operation uses bilinear interpolation; the convolution kernel size of the convolution layer of the final output result is 1×1 with a step size of 1.

[0053] Although Unet is a classic image segmentation model, it is not very suitable for tobacco leaf lesion segmentation. This application addresses the problems of large differences in lesion morphology and blurred edges in lesion segmentation, and designs a semantic segmentation model based on Unet consisting of two stacked sub-networks.

[0054] S22: Construct a first sub-network.

[0055] The first sub-network is improved based on the Unet network. The 9 blocks of the Unet network are replaced by 9 MC modules, a CBAM module is inserted after each MC module, and the AG (AttentionGate) attention gating mechanism is introduced in four jump connections. At the same time, to compensate for the lost detail information of the network, the RUFF (Feature Fusion Method Based on Residual Paradigm) module is used in the upsampling stage.

[0056] like Figure 4 As shown, the MC module structure is as follows Figure 4 The figure shows that it can be divided into two parts. The first part is a 3×3 convolution, and the second part is composed of 1×1, 3×3, and 5×5 convolutions in parallel. Residual connections are introduced to prevent the network from being too deep, which may cause gradient vanishing or gradient explosion. It is an efficient convolution block that can obtain target feature information of different scales. The specific process of the MC module processing the input feature map is as follows:

[0057] (1) The input feature map of size H×W×C is fed into a 3×3 convolution for convolution operation, and then batch normalization operation and ReLU activation function operation are performed in sequence through the BN layer to obtain an output feature map of size H×W×C / 2.

[0058] (2) The output results in step (1) are further extracted by 1×1, 3×3, and 5×5 convolutions, and batch normalization and ReLU activation functions are performed in turn through the BN layer to obtain output feature maps of sizes H×W×C / 4, H×W×C / 2, and H×W×C / 4, respectively.

[0059] (3) The three output results in step (2) are concatenated and fused in the channel dimension to obtain a feature map of size H×W×C.

[0060] (4) The input feature map of size H×W×C in step (1) is passed through a 1×1 convolutional layer and a BN layer as the residual connection of the MC module.

[0061] (5) The output results in step (3) and the output results in step (4) are added element by element, and then activated by the ReLU activation function to obtain the final output feature map of the MC module with a size of H×W×C.

[0062] like Figure 5 As shown in Figure 1, the CBAM module consists of a channel attention module and a spatial attention module in series. It can model the importance of each channel and each spatial position and focus on key information. The specific process of the CBAM module processing the input feature map is as follows:

[0063] (1) Perform global average pooling and global maximum pooling operations on the input feature map of size H×W×C respectively to obtain two feature maps of size 1×1×C.

[0064] (2) The two feature maps in step (1) are respectively sent to a shared two-layer perceptron. The first fully connected layer reduces the number of channels to the original 1 / r, and the second fully connected layer restores the number of channels to C. The ReLU function is used for activation between the two fully connected layers.

[0065] (3) Add the two feature maps obtained in step (2) and then pass them through a Sigmoid activation function to obtain a channel attention weight of size 1×1×C.

[0066] (4) Multiply the channel attention weights obtained in step (3) by the input feature map in step (1) channel by channel.

[0067] (5) Perform global average pooling and global maximum pooling operations on the results in step (4) in the channel dimension to obtain two feature maps of size H×W×1.

[0068] (6) Concatenate the two feature maps in step (5) and then reduce the number of channels to 1 through a 7×7 convolutional layer to obtain the spatial attention weight map.

[0069] (7) Multiply the spatial attention weight map obtained in step (6) with the feature map obtained in step (4) position by position to obtain the final output result of the CBAM module.

[0070] The AG module is a general self-attention mechanism that can eliminate redundant information and achieve accurate feature screening. The ReLU activation function in the AG attention gating mechanism is replaced by the LeakyReLU activation function to alleviate the neuron death problem and expand the function range. The AG module structure is as follows Figure 6 As shown in Figure 2, the specific process of the AG module processing the input feature map is as follows:

[0071] (1) Perform a 1×1 convolution on the input g of size H×W×F_g to obtain a feature map g1 of size H×W×F_int.

[0072] (2) Perform a 1×1 convolution on the input x from the skip connection of size H×W×F_l to obtain a feature map x1 of size H×W×F_int.

[0073] (3) Add g1 and x1 element by element, and then activate them through the LeakyReLU function.

[0074] (4) The feature map obtained in step (3) is subjected to 1×1 convolution to obtain a feature map of size H×W×1, and then batch normalization and Sigmoid activation function are applied in sequence to obtain the attention weight.

[0075] (5) Multiply the attention weights obtained in step (4) by the feature map x element-wise to obtain the final output of the AG module.

[0076] The RUFF module compensates for the information loss during the resolution restoration process and corrects the dislocation of spatial information through information fusion. The specific implementation process of the RUFF module is as follows:

[0077] (1) The outputs of the last CBAM module in the two sub-network encoders and the first three CBAMs in the decoder are respectively sent to the feature alignment module BU (Feature Alignment Module) for processing.

[0078] (2) The four output feature maps obtained in step (1) are channel-wise spliced and then processed sequentially through 3×3 convolution, BN layer, ReLU activation function, and CBAM module.

[0079] (3) Add the output result in step (2) and the output of the last CBAM module of the sub-network element by element to obtain the final output feature map of the module.

[0080] Furthermore, the structure of the BU module is Conv2d+BN+ReLU+Upsample, and the convolution kernel size is 1×1. In the first subnetwork, the 1×1 convolution reduces the number of channels of the input feature map to 8 channels, and in the second subnetwork, the 1×1 convolution reduces the number of channels of the input feature map to 16 channels; the Upsample method is bilinear interpolation.

[0081] S23, construct the module of interest ROIE+ (Modules of interest), which processes the output of the first sub-network. The ROIE+ module is located between the two sub-networks and is used to enhance the region of interest of the input image, suppress useless background, and improve the input image quality of the second sub-network. Its structure is as follows: Figure 7 shown.

[0082] The ROIE+ module of this embodiment adds two learnable weight parameters α and β initialized to 1 based on the ROIE module proposed by Liu. The specific process of the ROIE+ module processing the input feature map is as follows:

[0083] (1) Perform a dot product of the input x1 and output u of the first sub-network, and then multiply it by the weight α.

[0084] (2) The result obtained in step (1) and the result of multiplying the input x1 of the first sub-network by the weight β are added together as the final output of the ROIE+ module.

[0085] S24: Build a second sub-network, and use the output of the ROIE+ module in S23 as the input of the second sub-network.

[0086] The structure of the second sub-network is basically the same as that of the first sub-network in S22. The difference is that a dense residual dilated convolution module MVCR (Dense Residual Dilated Convolution Module) is embedded on the basis of the first sub-network; preferably, this embodiment embeds the MVCR module between the last MC module + CBAM module in the encoder and the first MC module + CBAM module in the decoder.

[0087] The MVCR module is designed with dilated convolution and has four branches. Each branch has a receptive field of different sizes, which can handle multiple types of targets and capture a wider range of contextual semantic information. Its structure is as follows: Figure 8 shown.

[0088] The four branches of the MVCR module are: four 3×3 convolutions with dilation rates of 1, 2, 5, and 8 are sequentially connected as the first branch, three 3×3 convolutions with dilation rates of 1, 2, and 5 are sequentially connected as the second branch, two 3×3 convolutions with dilation rates of 1 and 2 are sequentially connected as the third branch, and one 3×3 convolution with dilation rate of 1 is the fourth branch; each convolution is followed by a BN layer for batch normalization and a ReLU activation function for activation. Residual connections are introduced in each branch to ensure information flow and avoid network degradation. The specific process of the MVCR module processing the input feature map is as follows:

[0089] (1) The input feature map of size 32×32×512 is sent to the four branches for calculation respectively, and the output feature map of each branch is still of size 32×32×512.

[0090] (2) The output feature maps of the four branches in step (1) are concatenated and fused in the channel dimension to obtain a feature map of size 32×32×2048.

[0091] (3) Use 1×1 convolution to reduce the channel dimension of the output feature map in step (2) and restore the feature map size to 32×32×512.

[0092] (4) The output results in step (3) are processed by the BN layer, ReLU activation function, and Dropout layer (which randomly shuts down some neurons during the training process, reducing the complexity of the network and helping to prevent overfitting) to obtain the final output feature map of the MVCR module with a size of 32×32×512.

[0093] Step S25: Send the output of the encoder part of the first sub-network to the second sub-network for feature multiplexing.

[0094] Specifically, the feature reuse process is as follows: the outputs of the first, second, third, and fourth MC modules of the second sub-network are concatenated with the outputs of the first, second, third, and fourth CBAM modules of the first sub-network in the channel dimension, and then sent to the first, second, third, and fourth CBAM modules of the second sub-network respectively.

[0095] Step S26: Obtain a semantic segmentation model based on multi-scale residual dilated convolution.

[0096] On the basis of the Unet network, the semantic segmentation model based on multi-scale residual dilated convolution is constructed by replacing each block with the MC module, inserting the attention mechanism, and using the feature fusion method based on the residual paradigm. The overall structure is as follows: Figure 2 The algorithm structure is shown in Table 1.

[0097] Table 1: Algorithm structure of semantic segmentation model based on multi-scale residual dilated convolution in this application

[0098]

[0099]

[0100]

[0101]

[0102] Step S103: Combine the training set and use the constructed loss function to train the semantic segmentation model.

[0103] Specifically, the sample images in the training set are input into a pre-built semantic segmentation model for prediction to generate a semantic segmentation result of the sample image, which identifies the area of tobacco leaf spots in the sample image in a pixel-level manner; a loss function is constructed based on the semantic segmentation result of the sample image and the labeling result of the sample image, and based on the loss function, the back propagation algorithm is used to update the parameters of the semantic segmentation model to obtain a trained semantic segmentation model.

[0104] The loss function consists of the joint loss function of Dice Loss and Focal Loss, which is defined as follows:

[0105] Loss = αL Dice +βL Focal

[0106] Among them, L Dice is the Dice Loss function, L Focal is the Focal Loss loss function, α and β are the corresponding weight coefficients, both are 1.

[0107] The Dice Loss loss function is defined as follows:

[0108]

[0109] Among them, y i and are the true value and predicted value of pixel i, respectively, and N is the total number of pixels.

[0110] The Focal Loss loss function is defined as follows:

[0111] L Focal =-α t (1-p t ) γ log(p t)

[0112] Among them, p t is the probability that the model predicts a positive sample, α t is a balancing factor used to balance the weights of positive and negative samples to solve the problem of sample imbalance, (1-p t ) γ It is a regulation factor that increases the attention to difficult-to-classify samples by adjusting the size of γ.

[0113] Step S104: Use the validation set to validate the trained semantic segmentation model obtained in step S103, and select the optimal semantic segmentation model from the trained semantic segmentation models; the validation set is 200 sample images in the self-built dataset obtained in step S101.

[0114] Step S105: Use a test set to test the optimal semantic segmentation model obtained in step S104, and use evaluation indicators to evaluate the performance of the optimal semantic segmentation model; the test set is 200 sample images in the self-built dataset obtained in step S101.

[0115] S51. Evaluation index screening. This embodiment uses five evaluation indicators: Class Pixel Accuracy (CPA), Recall, Intersection over Union (IoU), Mean Intersection over Union (MIoU), F1 score, and Dice coefficient. The calculation methods are as follows:

[0116]

[0117] Here, TP represents the number of samples correctly predicted as positive; FP represents the number of samples incorrectly predicted as positive from the negative class; FN represents the number of samples incorrectly predicted as negative from the positive class; K represents the total number of classes; X and Y represent two sets (predicted values and true labels), |X∩Y| represents the number of elements in the intersection of the two sets, and |X| and |Y| represent the number of elements in the two sets. CPA calculates the proportion of pixels correctly classified for each class. Recall evaluates the model's ability to find all actual positive examples. IoU, MIoU, and Dice scores measure the similarity between pixel classification results and actual values, but are calculated slightly differently. The F1 score is the harmonic mean of precision and recall. All six evaluation criteria range from 0 to 1, with values closer to 1 indicating better model performance and closer to 0 indicating worse performance.

[0118] S52. Model evaluation results and analysis.

[0119] (1) Quantitative results analysis.

[0120] Unet, PSP, DeepLab v3+, FCN, SegNet, UNET++, DoubleU-Net, and the semantic segmentation model based on multi-scale residual dilated convolution in this application were tested on the test set data. The results of each evaluation indicator are shown in Table 2. Table 3 compares the GFLOPs and parameter data of each model.

[0121] Table 2 Evaluation results of each model

[0122]

[0123] Table 3 Comparison of GFLOPs and Parameters of each model

[0124] Model GFLOPs Parameters / M Unet 218.97 31.04 DoubleU-Net 215.84 29.29 UNET++ 249.78 67.98 PSP 262.74 65.70 DeepLab v3+ 86.90 54.70 FCN 102.00 18.64 SegNet 170.41 29.48 Model of the present invention 233.92 46.50

[0125] As can be seen from Table 2, the experimental results of DeepLab v3+ are relatively the worst, Unet is second best, and the semantic segmentation model based on multi-scale residual dilated convolution of the present invention is the best. Specifically, the semantic segmentation model based on multi-scale residual dilated convolution of the present invention is 14.9%, 9.36%, 18.71%, 12.44%, and 8.89% higher than DeepLab v3+ in lesion CPA, Recall, IoU, F1, and overall Dice score, respectively, and is 2.3%, 2.14%, 3.34%, 2%, and 1.46% higher than Unet, respectively.

[0126] As can be seen from Table 3, the semantic segmentation model based on multi-scale residual dilated convolution in the present invention ranks third and fourth in GFLOPs and Parameters, respectively, with moderate model complexity. In summary, the semantic segmentation model based on multi-scale residual dilated convolution in the present invention has the best overall performance.

[0127] (2) Qualitative results analysis.

[0128] Figure 10 The following are the recognition results of tobacco leaf lesions using Unet, PSP, DeepLab v3+, FCN, SegNet, UNET++, DoubleU-Net, and the semantic segmentation model based on multi-scale residual dilated convolutions of the present invention. As can be seen from the figure, DeepLabv3+ exhibits significant over-segmentation and under-segmentation in certain areas. Even the suboptimal Unet still suffers from problems such as unclear and incomplete lesion segmentation boundaries, which are sticky, unclear, and incomplete. However, the semantic segmentation model based on multi-scale residual dilated convolutions of the present invention can more accurately distinguish the boundaries of lesions of various sizes and shapes, achieving better results.

[0129] Step S106: Obtain the image to be segmented, input the image to be segmented into the trained semantic segmentation model, and obtain the semantic segmentation result of the image to be segmented.

[0130] Among them, the image to be segmented is a tobacco leaf disease image, and the semantic segmentation result is a tobacco leaf disease spot segmentation map.

[0131] Corresponding to the above method, this embodiment also provides a tobacco leaf spot segmentation system based on multi-scale residual dilated convolution, comprising:

[0132] The dataset construction module is used to collect multiple tobacco leaf disease images as original images, use image enhancement methods to expand the data of each collected original image, and establish training sets, validation sets and test sets based on the original images and expanded data.

[0133] The model building module is used to build a semantic segmentation model. The semantic segmentation model is composed of two stacked sub-networks, and the two sub-networks are connected by the ROIE+ module.

[0134] The model training and verification module is used to train the semantic segmentation model using the constructed loss function in combination with the training set; the trained semantic segmentation model is verified using the verification set to select the optimal semantic segmentation model.

[0135] The model evaluation module is used to test the optimal semantic segmentation model using the test set and evaluate the performance of the optimal semantic segmentation model.

[0136] The lesion segmentation module is used to input the tobacco leaf disease image to be segmented into the trained semantic segmentation model to obtain the tobacco leaf lesion segmentation map.

[0137] Unless otherwise specifically stated, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0139] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0140] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or software functional modules. The present invention is not limited to any specific combination of hardware and software.

[0141] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A tobacco leaf spot segmentation method based on multi-scale residual dilated convolution, characterized in that: Include: Collect multiple tobacco leaf disease images as original images, use image enhancement methods to expand the data of each collected original image, and establish training sets, validation sets, and test sets based on the original images and expanded data; Build a semantic segmentation model, which consists of two stacked sub-networks connected by the ROIE+ module; Combined with the training set, the constructed loss function is used to train the semantic segmentation model; the trained semantic segmentation model is verified using the validation set to select the optimal semantic segmentation model; Use the test set to test the optimal semantic segmentation model and evaluate its performance; The tobacco leaf disease image to be segmented is input into the trained semantic segmentation model to obtain the tobacco leaf disease spot segmentation map.

2. The tobacco leaf spot segmentation method based on multi-scale residual dilated convolution according to claim 1 is characterized in that: The first and second subnetworks of the semantic segmentation model are both improved from the Unet basic network framework. The 9 convolutional blocks of the Unet network are replaced by 9 MC modules, and a CBAM module is inserted after each MC module.

3. The tobacco leaf spot segmentation method based on multi-scale residual dilated convolution according to claim 2, characterized in that: Both sub-networks introduce attention gating in each skip connection and use RUFF modules in the upsampling stage. The RUFF module implementation process is as follows: ①, the output of the last CBAM module in the encoder and the first three CBAM modules in the decoder of the two sub-networks are respectively fed into the feature alignment module for processing; ②, the four output feature maps obtained in ① are channel-wise concatenated and then processed sequentially through 3×3 convolution, BN layer, ReLU activation function, and CBAM module; ③. Add the output result of ② and the output of the last CBAM module of the sub-network element by element to obtain the final output feature map of the RUFF module.

4. The tobacco leaf spot segmentation method based on multi-scale residual dilated convolution according to claim 2, characterized in that: The MC module consists of two parts. The first part is a 3×3 convolution, and the second part is composed of 1×1, 3×3, and 5×5 convolutions in parallel, and a residual connection is introduced. The process of the MC module to process the input feature map is as follows: ①, the input feature map is sent to the first part of a 3×3 convolution for convolution operation to obtain the output feature map; ②, the second part of the 1×1, 3×3, and 5×5 convolutions are used to further extract features from the output of ①, respectively, to obtain three output feature maps of different sizes. ③. Concatenate and fuse the three output results of ② in the channel dimension; ④. Pass a 1×1 convolution on the input feature map of ① as the residual connection of the MC module; ⑤. Add the fusion result of ③ and the output result of ④ element by element to obtain the final output feature map of the MC module.

5. The tobacco leaf spot segmentation method based on multi-scale residual dilated convolution according to claim 2, characterized in that: The CBAM module consists of a channel attention module and a spatial attention module in series. The CBAM module processes the input feature map as follows: ① It performs global average pooling and global maximum pooling operations on the input feature map to obtain two output feature maps; ② It feeds the two output feature maps of ① into a shared two-layer perceptron. ③. Add the two feature maps after ② to get the channel attention weight; ④. Multiply the channel attention weight obtained in ③ by the input feature map in ① channel by channel; ⑤. Perform global average pooling and global maximum pooling operations on the results of ④ to obtain two output feature maps; ⑥. Concatenate the two output feature maps of ⑤ and then reduce the number of channels through a convolutional layer to obtain the spatial attention weight map. ⑦. Multiply the spatial attention weight map obtained by ⑥ with the feature map obtained by ④ position by position to obtain the final output result of the CBAM module.

6. The tobacco leaf lesion segmentation method based on multi-scale residual dilated convolution according to claim 2, characterized in that: An MVCR module is embedded in the second sub-network, and the output features of the encoder part of the first sub-network are reused. Feature reuse means that in the second sub-network, the output of the encoder MC module and the output of the CBAM module of the corresponding layer in the first sub-network are concatenated in the channel dimension and then sent to the CBAM module of the corresponding layer in the second sub-network.

7. The tobacco leaf lesion segmentation method based on multi-scale residual dilated convolution according to claim 6, characterized in that: The MVCR module is designed with dilated convolution and has a total of 4 branches. The 4 branches are: 4 3×3 convolutions with dilation rates of 1, 2, 5, and 8 are connected in series as the first branch, 3 3×3 convolutions with dilation rates of 1, 2, and 5 are connected in series as the second branch, 2 3×3 convolutions with dilation rates of 1 and 2 are connected in series as the third branch, and 1 3×3 convolution with dilation rate of 1 is used as the fourth branch. Residual connection is introduced in each branch. The process of MVCR module processing input feature maps is as follows: ①, the input feature map is sent to the 4 branches for calculation to obtain four output feature maps; ②, the four output feature maps of ① are spliced and fused in the channel dimension to obtain an output feature map; ③ 1×1 convolution is used to reduce the channel dimension of the output feature map of ②. ④. The output results of ③ are processed by the BN layer, ReLU activation function, and Dropout layer in sequence to obtain the final output feature map of the MVCR module.

8. The tobacco leaf lesion segmentation method based on multi-scale residual dilated convolution according to claim 2, characterized in that: The process of ROIE+ module processing the input feature map is as follows: ①, perform dot multiplication on the input and output of the first sub-network, and then multiply it by the weight α; ②, add the result obtained by ① and the result of multiplying the input of the first sub-network by the weight β as the final output of the ROIE+ module.

9. The tobacco leaf lesion segmentation method based on multi-scale residual dilated convolution according to claim 1, characterized in that: The loss function consists of the joint loss function of Dice Loss and Focal Loss, which is defined as follows: Loss=αL Dice +βL Focal Among them, L Dice is the Dice Loss function, L Focal is the Focal Loss loss function, and α and β are the corresponding weight coefficients.

10. A tobacco leaf spot segmentation system based on multi-scale residual dilated convolution, characterized by: Include: The dataset construction module is used to collect multiple tobacco leaf disease images as original images, perform data augmentation on each collected original image using image enhancement methods, and establish training sets, validation sets, and test sets based on the original images and augmented data; The model building module is used to build a semantic segmentation model. The semantic segmentation model is composed of two stacked sub-networks, which are connected by the ROIE+ module. The model training and verification module is used to train the semantic segmentation model using the constructed loss function in combination with the training set; the trained semantic segmentation model is verified using the verification set to select the optimal semantic segmentation model; The model evaluation module is used to test the optimal semantic segmentation model using the test set and evaluate the performance of the optimal semantic segmentation model; The lesion segmentation module is used to input the tobacco leaf disease image to be segmented into the trained semantic segmentation model to obtain the tobacco leaf lesion segmentation map.

Images