A method for identifying vegetation leaf diseases and their severity

Abstract

The present invention discloses a method for identifying plant leaf diseases and their severity. The method comprises a TSTC network, which includes a multi-task feature extraction module, a feature fusion module, and a deep supervision module. Leaf classification labels and a training set are preset. Leaf disease images in the training set are input into the multi-task feature extraction module, which outputs shallow features. The feature fusion module fuses the shallow features to obtain compact features. The shallow features and compact features are input into the deep supervision module, which processes the shallow features and obtains predicted values. The total loss of the network is calculated based on the predicted values ​​and the leaf classification labels. The network parameters are back-propagated and updated to obtain an updated TSTC network. The leaf disease image to be detected is input into the updated TSTC network, which processes the predicted values ​​and outputs the predicted values. The disease type and severity of the leaf to be detected are then obtained based on the predicted values. This method can simultaneously identify both the disease and severity of a plant leaf.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for identifying vegetation leaf diseases and their severity. Background Art

[0002] Vegetation is threatened by a variety of diseases during its growth, such as scab, leaf spot, and cedar rust. These diseases can affect the photosynthetic capacity of plant leaves, reducing plant quality and yield, and even potentially killing the plant. Timely and accurate detection of plant diseases facilitates treatment and prevents the spread of disease. Therefore, identifying plant leaf diseases is one of the most important issues in smart agriculture.

[0003] Over the past few decades, various methods have been used to identify plant diseases. Traditionally, the approach relies on observing leaf features, a time-consuming, labor-intensive process with low disease recognition accuracy. With the rapid development of computer vision, image processing techniques have been widely applied to plant disease recognition. Support vector machines, self-organizing feature maps, and nearest neighbor classification have demonstrated excellent performance, but these classifiers are often constructed based on handcrafted features. Studies have shown that handcrafted features have limited expressive power and struggle to distinguish subtle differences in complex disease signatures. In recent years, deep learning has achieved promising results in image classification tasks due to its powerful feature extraction and pattern recognition capabilities. Convolutional neural networks (CNNs), a typical deep learning network, are particularly well-suited for agricultural disease recognition due to their flexible topology and the ability to employ demanding feature representations for complex diseases. CNNs, for example, have been widely used in the field of agricultural disease recognition. For example, Amara et al. used a deep learning network based on the LeNet architecture to classify banana leaf diseases, while Ferentinos employed VGG and AlexNet / OWTBn networks for plant disease detection. However, CNNs have limited receptive fields and are difficult to capture global information, which has certain limitations in disease recognition.

[0004] In addition, the vegetation disease recognition model is basically a single-task model, which takes vegetation type-disease or vegetation type-disease-severity as a category. When a new class needs to be added, the model still needs to be retrained even if the vegetation type and its disease information already exist in the previous dataset. Summary of the Invention

[0005] In view of the shortcomings of existing vegetation disease identification methods, the present invention proposes a method for identifying vegetation leaf diseases and their severity, which includes:

[0006] S1. Build a TSTC network. The TSTC network includes a multi-task feature extraction module, a feature fusion module, and a deep supervision module connected in sequence.

[0007] S2, presetting leaf classification labels and training sets, inputting leaf disease images in the training set into the multi-task feature extraction module of the TSTC network, and obtaining three sets of shallow features after processing;

[0008] The feature fusion module of S3 and TSTC networks uses compact bilinear pooling technology to fuse three sets of shallow features to obtain two sets of compact features;

[0009] S4. Input the three sets of shallow features and two sets of compact features into the deep supervision module of the TSTC network. After processing, the corresponding prediction values ​​are obtained. The total loss of the TSTC network is calculated based on the prediction values ​​and the leaf classification labels. The network parameters of the network are updated through backpropagation to obtain the updated TSTC network.

[0010] S5. Input the diseased image of the leaf to be detected into the updated TSTC network, output the predicted value of the leaf to be detected after processing, and obtain the type and severity of the disease of the leaf to be detected based on the predicted value of the leaf to be detected.

[0011] Preferably, the multi-task feature extraction module in S1 includes a blocking module and several task feature extraction sub-modules, each task feature extraction sub-module includes a linear embedding module or a blocking merging module, and several Swin Transformer block modules, the linear embedding module or the blocking merging module is connected to several Swin Transformer block modules, the blocking module is used to block the input leaf disease image, and the several multi-task feature extraction sub-modules are divided into four stages and three branch networks to extract the shallow features of the leaf disease image after blocking.

[0012] Preferably, S2 inputs the leaves in the training set into the multi-task feature extraction module of the TSTC network, and obtains three sets of shallow features after processing, specifically including:

[0013] S21, inputting the leaf disease image in the training set into the block module in the multi-task feature extraction module, and outputting a plurality of block leaf disease images after processing;

[0014] S22, using the first-stage network and the second-stage network of the multi-task feature extraction module to extract shallow shared features of multiple block-based leaf disease images;

[0015] S23, using the third-stage network of the multi-task feature extraction module and the first branch in the fourth-stage network to identify shallow shared features, and output a first shallow feature f1 for the disease type;

[0016] S24, using the third-stage network of the multi-task feature extraction module and the second branch in the fourth-stage network to identify shallow shared features and output a second shallow feature f2 for disease severity;

[0017] S25. Use the third-stage network of the multi-task feature extraction module and the third branch in the fourth-stage network to identify shallow shared features and output the third shallow feature f3 for supervising the TSTC network.

[0018] Preferably, the feature fusion module in S1 includes a first CBP layer and a second CBP layer connected in parallel. The feature fusion module in S3 uses a compact bilinear pooling technique to fuse the three sets of shallow features to obtain two sets of compact features, specifically including:

[0019] S31, input the first shallow feature f1 and the third shallow feature f3 into the first CBP layer, and fuse them using compact bilinear pooling technology to generate the first compact feature f 13 ;

[0020] S32, input the second shallow feature f2 and the third shallow feature f3 into the second CBP layer, and fuse them using compact bilinear pooling technology to generate the second compact feature f 23 .

[0021] Preferably, the first compact feature and the second compact feature can be expressed by the formula:

[0022]

[0023]

[0024] Where B(·) represents bilinear features, C(·) represents compact features, φ(·) represents low-dimensional projection function, S, U, and K represent positions, f1, f2, and f3 represent the first, second, and third shallow features, respectively, and f13 and f23 are the first and second compact features, respectively.

[0025] Preferably, the deep supervision module in S1 includes several Flatten layers and fully connected layers (FC). S4 inputs three sets of shallow features and two sets of compact features into the deep supervision module, and outputs corresponding prediction values ​​after processing, specifically including:

[0026] S41: Input the first shallow feature f1 to the first Flatten layer and the first fully connected layer connected in sequence, and output the disease classification prediction value after processing.

[0027] S42, the second compact feature f 23 The input is sent to the second and third fully connected layers connected in sequence, and the first severity classification prediction value is output after processing.

[0028] S43, the first compact feature f 13 Input to the fourth and fifth fully connected layers connected in sequence, and after processing, output the first overall classification prediction value

[0029] S44: Input the second shallow feature f2 to the second Flatten layer and the sixth fully connected layer connected in sequence, and output the second overall classification prediction value after processing.

[0030] S45: Input the third shallow feature f3 to the third Flatten layer and the seventh fully connected layer connected in sequence, and output the second severity classification prediction value after processing.

[0031] Preferably, in S4, the total loss of the TSTC network is calculated according to the predicted value and the leaf classification label, specifically including:

[0032] S46, using a softmax classifier to normalize the classification prediction value, and correspondingly obtaining a normalized classification prediction value of the leaf image;

[0033] S47, calculating the leaf classification loss according to the normalized classification prediction value and the leaf classification label;

[0034] S48. Calculate the intermediate cross entropy loss and the output cross entropy loss according to the leaf classification loss;

[0035] S49. Calculate the total loss of the TSTC network based on the sum of the intermediate cross entropy loss and the output cross entropy loss.

[0036] Preferably, S47 calculates the leaf classification loss based on the normalized classification prediction value and the leaf classification label. The specific formula is:

[0037]

[0038] Where, l i is the i-th classification loss of leaf, is the normalized predicted value p of the i-th leaf i The mth component in , is the leaf classification label corresponding to the mth component of the i-th classification, and n is the total number of components in the predicted value.

[0039] Preferably, the calculation formula of the intermediate cross entropy loss in S48 is specifically:

[0040]

[0041] The calculation formula of the output cross entropy loss in S48 is as follows:

[0042]

[0043] Where, L s is the intermediate cross entropy loss, l3 is the first overall classification loss, l4 is the second overall classification loss, l5 is the second severity classification, L o is the output cross entropy loss, l1 is the disease classification loss, l2 is the first severity classification loss, Represents the weight corresponding to the i-th classification loss (i=1,2,3,4,5).

[0044] Preferably, the predicted value includes a disease classification predicted value and a first severity classification predicted value. In S5, the disease type and severity of the leaf to be detected are obtained according to the predicted value of the leaf to be detected, specifically including:

[0045] S51, obtaining the disease type of the leaf to be detected according to the disease classification prediction value of the leaf to be detected;

[0046] S52: Obtain the severity of the leaf to be detected according to the first severity classification prediction value of the leaf to be detected.

[0047] The above-mentioned method for identifying vegetation leaf diseases and their severity uses an end-to-end trainable TSTC network, which can autonomously extract required features through learning during training. A three-branch Swin Transformer network is proposed in the multi-task feature extraction module of the TSTC network, which not only realizes multi-task classification but also effectively captures global information and has a wide receptive field. A compact bilinear pooling technique is used to fuse features, making the features more discriminative. A deep supervision module is designed to enhance the feature extraction capability of the hidden layer, thereby improving the TSTC network's recognition performance for vegetation diseases and their severity. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of a method for identifying vegetation leaf diseases and their severity in one embodiment of the present invention;

[0049] Figure 2 is an overall block diagram of a TSTC network in one embodiment of the present invention;

[0050] Figure 3 is a structural block diagram of a multi-task feature extraction module in one embodiment of the present invention;

[0051] Figure 4 is a schematic structural diagram of a Swin Transformer block module in one embodiment of the present invention;

[0052] Figure 52 is a block diagram of the structure of the deep supervision module in one embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0054] A method for identifying vegetation leaf diseases and their severity, comprising:

[0055] S1. Build a TSTC network. The TSTC network includes a multi-task feature extraction module, a feature fusion module, and a deep supervision module connected in sequence.

[0056] S2, presetting leaf classification labels and training sets, inputting leaf disease images in the training set into the multi-task feature extraction module of the TSTC network, and obtaining three sets of shallow features after processing;

[0057] The feature fusion module of S3 and TSTC networks uses compact bilinear pooling technology to fuse three sets of shallow features to obtain two sets of compact features;

[0058] S4. Input the three sets of shallow features and two sets of compact features into the deep supervision module of the TSTC network. After processing, the corresponding prediction values ​​are obtained. The total loss of the TSTC network is calculated based on the prediction values ​​and the leaf classification labels. The network parameters of the network are updated through backpropagation to obtain the updated TSTC network.

[0059] S5. Input the diseased image of the leaf to be detected into the updated TSTC network, output the predicted value of the leaf to be detected after processing, and obtain the type and severity of the disease of the leaf to be detected based on the predicted value of the leaf to be detected.

[0060] Specifically, see Figure 1 and Figure 2 , Figure 1 This is a flow chart of a method for identifying vegetation leaf diseases and their severity in one embodiment of the present invention. Figure 2 FIG. 4 is an overall block diagram of a TSTC network in one embodiment of the present invention.

[0061] A method for identifying vegetation leaf diseases and their severity is first constructed. The TSTC network (triple-branch Swin Transformer classification network) is mainly composed of three modules: a multi-task feature extraction module, a feature fusion module, and a deep supervision module. Then, the leaf classification labels and training sets are set, and the leaf disease images in the training set are input into the multi-task feature extraction module to extract three groups of shallow features specifically for disease and severity classification tasks. The three groups of shallow features are input into the feature fusion module, which uses the compact bilinear pooling technology (CBP) to further fuse these shallow features to obtain two groups of compact features, thereby enhancing the feature expression ability. The three groups of shallow features and the two groups of compact features are input into the deep supervision module, which helps the TSTC network The hidden layer of the network (the hidden layer refers to the middle layer except the input layer and the output layer in the entire TSTC network) extracts more discriminative features, and after processing, the corresponding prediction value is output; then the softmax classifier is used to normalize the prediction value to obtain the normalized prediction value, and the total loss of the TSTC network is calculated according to the normalized prediction value and the preset leaf classification label, and the network parameters of the network are updated by back propagation to obtain the updated TSTC network; finally, the leaf disease image to be detected is input into the updated TSTC network, and after processing, the prediction value of the leaf to be detected is output, and the type and severity of the disease of the vegetation leaf to be detected are obtained according to the prediction value of the leaf to be detected.

[0062] In one embodiment, the multi-task feature extraction module in S1 includes a blocking module and several task feature extraction sub-modules, each task feature extraction sub-module includes a linear embedding module or a blocking merging module, and several Swin Transformer block modules, the linear embedding module or the blocking merging module is connected to several Swin Transformer block modules, the blocking module is used to block the input leaf disease image, and the several task feature extraction sub-modules are divided into four stages and three branch networks to extract shallow features of the leaf disease image.

[0063] For details, see Figure 3 , Figure 3 FIG. 4 is a structural block diagram of a multi-task feature extraction module in one embodiment of the present invention.

[0064] The multi-task feature extraction module uses Swin Transformer as the backbone and constructs a three-branch Swin Transformer network to extract multi-task classification features from the input image, where one branch is used to identify the disease, one branch is used to identify the severity, and the other branch is used for deep supervision.

[0065] exist Figure 3 In [1], the multi-task feature extraction module includes a block module and eight task feature extraction submodules connected sequentially or in parallel. The eight task feature extraction submodules are divided into four stages and three branches to extract task features. Each stage task feature extraction submodule includes a block merging module or a linear embedding module, and several SwinTransformer block modules. The specific network structure of the multi-task feature extraction module is as follows:

[0066] The first-stage network is the first-task feature extraction submodule, which includes a linear embedding module and a Swin Transformer block module, where the Swin Transformer block module is connected to the linear embedding module;

[0067] The second-stage network is the second-task feature extraction submodule, which is connected to the first-task feature extraction submodule. The second-task feature extraction submodule includes a block merging module and a SwinTransformer block module, where the Swin Transformer block module is connected to the block merging module;

[0068] The third-stage network is divided into three branches. The first, second, and third branches are the third, fourth, and fifth task feature extraction submodules respectively. The third, fourth, and fifth task feature extraction submodules are all connected to the second task feature extraction submodule. The third, fourth, and fifth task feature extraction submodules each include a block merging module and three Swin Transformerblock modules, where the three Swin Transformerblock modules are connected to the block merging module in sequence.

[0069] The fourth-stage network is also divided into three branches. The first, second, and third branches are the sixth, seventh, and eighth task feature extraction sub-modules respectively. The sixth, seventh, and eighth task feature extraction sub-modules are connected to the third, fourth, and fifth task feature extraction sub-modules respectively. The sixth, seventh, and eighth task feature extraction sub-modules each include a block merging module and a Swin Transformer block module, in which the Swin Transformer block module is connected to the block merging module.

[0070] These eight task feature extraction submodules are divided into four stages and three branches to extract task features. The first and second task feature extraction submodules in the first and second stages are used to extract shallow shared features, and the shallow shared features are input into the third to eighth task feature extraction submodules in the third and fourth stages. Through the three branches composed of the third to eighth task feature extraction submodules, three groups of shallow features with specific classification tasks are obtained.

[0071] In one embodiment, S2 inputs the leaves in the training set into the multi-task feature extraction module of the TSTC network, and obtains three sets of shallow features after processing, specifically including:

[0072] S21, inputting the leaf disease image in the training set into the block module in the multi-task feature extraction module, and outputting a plurality of block leaf disease images after processing;

[0073] S22, using the first-stage network and the second-stage network of the multi-task feature extraction module to extract shallow shared features of multiple block-based leaf disease images;

[0074] S23, using the third-stage network of the multi-task feature extraction module and the first branch in the fourth-stage network to identify shallow shared features, and output a first shallow feature f1 for the disease type;

[0075] S24, using the third-stage network of the multi-task feature extraction module and the second branch in the fourth-stage network to identify shallow shared features and output a second shallow feature f2 for disease severity;

[0076] S25. Use the third-stage network of the multi-task feature extraction module and the third branch in the fourth-stage network to identify shallow shared features and output the third shallow feature f3 for supervising the TSTC network.

[0077] Specifically, see Figure 3In the multi-task feature extraction module, the vegetation leaf disease image is first divided into non-overlapping blocks by the blocking module, and then these blocks are input into the first and second stage networks. The features of the leaf disease image are extracted by the first and second task feature extraction submodules in the first and second stage networks. Since the first and second stage networks are shallow networks, the features extracted from the shallow networks are relatively few and are shallow shared features (the number of layers of the network is relatively shallow, and the extracted features are generally surface features of the vegetation leaf disease image, which contain relatively little information). The third and fourth stage networks are deep networks, and the features obtained from the deep networks are more abstract and contain rich semantic information of specific categories. The shallow shared features obtained from the first two stages are further fed into the three branches of the third and fourth stages to obtain deep features for specific classification tasks. In the third and fourth stages, the third and sixth task feature extraction submodules constitute the first branch, the fourth and seventh task feature extraction submodules constitute the second branch, and the fifth and eighth task feature extraction submodules constitute the third branch. The first branch identifies shallow shared features of leaf disease images and outputs a first shallow feature f1, which is used for disease type. The second branch identifies shallow shared features of leaf disease images and outputs a second shallow feature f2, which is used for disease severity. The third branch identifies shallow shared features of leaf disease images and outputs a third shallow feature f3, which is used to supervise the entire TSCT network. In the task feature extraction submodule, after each pass through the block merging module, the width and length of the features are halved, and the number of channels is doubled, gradually increasing the information receptive field and facilitating the acquisition of global information features.

[0078] In one embodiment, the Swin Transformer block module in the task feature extraction submodule includes a first layernorm layer (LN1), a window-based multi-head attention module (W-MSA), a second layernorm layer (LN2), a first multilayer perceptron (MLP1), a third layernorm layer (LN3), a moving window-based multi-head attention module (SW-MSA), a fourth layernorm layer (LN4) and a second multilayer perceptron (MLP2) connected in sequence.

[0079] Specifically, see Figure 4 , Figure 4 This is a structural diagram of the Swin Transformer block module.

[0080] The feature extraction process of the Swin Transformer block module can be expressed as follows:

[0081] feat 2 =W-MSA(LN(feat1 ))+feat 1

[0082] feat 3 =MLP(LN(feat 2 ))+feat 2

[0083] feat 4 =SW-MSA(LN(feat 3 ))+feat 3

[0084] feat 5 =MLP(LN(feat 4 ))+feat 4

[0085] In the formula, feat 1 Represents the input features of the Swin Transformer block module, feat 5 Represents the output features of the SwinTransformer block module, feat 2 、feat 3 、feat 4 The features extracted from the middle layer of the Swin Transformer block module are intermediate variables.

[0086] by Figure 3 For example, in the sixth task feature extraction submodule, the SwinTransformer block module receives features (corresponding to feat 1 ), after processing, the first shallow feature f1 is output (corresponding to feat 5 ).

[0087] In one embodiment, the feature fusion module in S1 includes a first CBP layer and a second CBP layer connected in parallel. The feature fusion module in S3 uses a compact bilinear pooling technique to fuse the three sets of shallow features to obtain two sets of compact features, specifically including:

[0088] S31, input the first shallow feature f1 and the third shallow feature f3 into the first CBP layer, and fuse them using compact bilinear pooling technology to generate the first compact feature f 13 ;

[0089] S32, input the second shallow feature f2 and the third shallow feature f3 into the second CBP layer, and fuse them using compact bilinear pooling technology to generate the second compact feature f 23 .

[0090] Specifically, the feature fusion module includes a first CBP layer and a second CBP layer in parallel. The feature fusion module uses the compact bilinear pooling technology CBP to fuse the three sets of shallow features f1, f2 and f3 output by the multi-task feature extraction module. In the feature fusion module, f1 and f3 are input into the first CBP layer to generate the fused feature f 13 , input f2 and f3 into the second CBP layer to generate fusion feature f 23 In general, severity classification is more challenging than disease type classification because different diseases may be identified as different severity levels. The fusion of f2 and f3 exploits the complementarity of the two features, making the extracted features more discriminative and generating the fused feature f 23 For the classification of disease types, the CBP technology is not used because the f1 extracted from the multi-task feature extraction module is usually sufficiently discriminative for the disease classification task and can well distinguish the types of vegetation leaf diseases. In addition, the CBP technology is used to fuse f1 and f3 to generate the fused feature f 13 , used for deep supervision of the entire TSTC network.

[0091] In one embodiment, the first compact feature and the second compact feature can be expressed as:

[0092]

[0093]

[0094] Where B(·) represents bilinear features, C(·) represents compact bilinear features, φ(·) represents low-dimensional projection function, S, U and K represent positions, f1, f2 and f3 represent the first, second and third shallow features respectively, and f 13 and f 23 The first and second compact features respectively.

[0095] Different from the traditional method of combining two sets of features, the feature fusion module adopts compact bilinear pooling technology (CBP): first, the high-dimensional features of different classification tasks are converted from high-dimensional space to low-dimensional space using projection technology to reduce calculation and storage space, and then the projected features are combined to generate compact features.

[0096] In one embodiment, the deep supervision module in S1 includes several Flatten layers and fully connected layers (FC). S4 inputs three sets of shallow features and two sets of compact features into the deep supervision module, and outputs corresponding prediction values ​​after processing, specifically including:

[0097] S41: Input the first shallow feature f1 to the first Flatten layer and the first fully connected layer connected in sequence, and output the disease classification prediction value after processing.

[0098] S42, the second compact feature f 23 The input is sent to the second and third fully connected layers connected in sequence, and the first severity classification prediction value is output after processing.

[0099] S43, the first compact feature f 13 Input to the fourth and fifth fully connected layers connected in sequence, and after processing, output the first overall classification prediction value

[0100] S44: Input the second shallow feature f2 to the second Flatten layer and the sixth fully connected layer connected in sequence, and output the second overall classification prediction value after processing.

[0101] S45: Input the third shallow feature f3 to the third Flatten layer and the seventh fully connected layer connected in sequence, and output the second severity classification prediction value after processing.

[0102] Specifically, see Figure 5 , Figure 5 2 is a structural block diagram of a deep supervision module in one embodiment of the present invention.

[0103] The deep supervision module consists of five branches through three Flatten layers and seven fully connected layers (FC) to predict the type and severity of the disease. The network structure is as follows:

[0104] The first branch of the deep supervision module includes the first Flatten layer and the first fully connected layer connected in sequence. The first shallow feature f1 (that is, the disease classification feature) is input into the first branch and the disease classification prediction value is obtained after processing.

[0105] The second branch of the deep supervision module includes the second fully connected layer and the third fully connected layer connected in sequence, which transforms the second compact feature f 23 Input the second branch and get the first severity classification prediction value after processing

[0106] The third branch of the deep supervision module includes the fourth fully connected layer and the fifth fully connected layer connected in sequence, which transforms the first compact feature f 13 Enter the third branch and get the first overall classification prediction value after processing

[0107] The fourth branch of the deep supervision module includes the second Flatten layer and the sixth fully connected layer connected in sequence. The second shallow feature f2 (that is, the disease severity classification feature) is input into the fourth branch and processed to obtain the second overall classification prediction value.

[0108] The fifth branch of the deep supervision module includes the third Flatten layer and the seventh fully connected layer connected in sequence. The third shallow feature f3 is input into the fifth branch and processed to obtain the second severity classification prediction value.

[0109] In one embodiment, S4 calculates the total loss of the TSTC network based on the predicted value and the leaf classification label, specifically including:

[0110] S46, using a softmax classifier to normalize the classification prediction value, and correspondingly obtaining a normalized classification prediction value of the leaf image;

[0111] S47, calculating the leaf classification loss according to the normalized classification prediction value and the leaf classification label;

[0112] S48. Calculate the intermediate cross entropy loss and the output cross entropy loss according to the leaf classification loss;

[0113] S49. Calculate the total loss of the TSTC network based on the sum of the intermediate cross entropy loss and the output cross entropy loss.

[0114] In one embodiment, S47 calculates the leaf classification loss based on the normalized classification prediction value and the leaf classification label. The specific formula is:

[0115]

[0116] Where, l i is the i-th classification loss of leaf, is the normalized predicted value p of the i-th leaf i The mth component in , is the leaf classification label corresponding to the mth component of the i-th classification, and n is the total number of components in the predicted value.

[0117] Since each prediction value output by each branch of the deep supervision module is a vector, for example, the normalized prediction value p of the i-th leaf disease i The corresponding vector has n=3 components, that is, p i =[0.2,0.3,0.5], at this time, the normalized predicted value p of the i-th i The three components in

[0118] In one embodiment, the calculation formula of the intermediate cross entropy loss in S48 is specifically:

[0119]

[0120] The calculation formula of the output cross entropy loss in S48 is as follows:

[0121]

[0122] Where, L s is the intermediate cross entropy loss, l3 is the first overall classification loss, l4 is the second overall classification loss, l5 is the second severity classification, L o is the output cross entropy loss, l1 is the disease classification loss, l2 is the first severity classification loss, Represents the weight corresponding to the i-th classification loss (i=1,2,3,4,5).

[0123] Specifically, the leaf disease images in the training set are input into the built TSTC network, and after processing, the corresponding output classification prediction value The total loss of the TSTC network is calculated based on the classification prediction value. The process is as follows:

[0124] 1) Use softmax classifier to classify the predicted value Normalization, the corresponding classification prediction value p of the leaf image after normalization is obtained i (i=1,2,3,4,5):

[0125] 2) Calculate the leaf classification loss based on the normalized classification prediction value and the leaf classification label. The specific formula is:

[0126]

[0127] Where, l i is the i-th classification loss of leaf, is the normalized predicted value p of the i-th leaf i The mth component in , is the leaf classification label corresponding to the mth component of the i-th classification, and n is the total number of components in the predicted value.

[0128] The first severity classification label and the second severity classification label may be the same, both corresponding to the severity classification label, and the first overall classification label and the second overall classification label may be the same, both corresponding to the overall classification label.

[0129] 3) Calculate the intermediate cross entropy loss and output cross entropy loss based on the leaf classification loss. The specific formula is:

[0130]

[0131]

[0132] Where, L s is the intermediate cross entropy loss, L o is the output cross entropy loss, l1 is the disease classification loss, l2 is the first severity classification loss, l3 is the first overall classification loss, l4 is the second overall classification loss, l5 is the second severity classification loss, Represents the weight corresponding to the i-th classification loss, i = 1, 2, 3, 4, 5.

[0133] 4) Calculate the total loss of the TSTC network based on the sum of the intermediate cross entropy loss and the output cross entropy loss.

[0134] Under the guidance of the joint loss of intermediate cross entropy loss and output cross entropy loss, the features extracted in the hidden layer are made more discriminative, which can improve the training accuracy of the TSTC network.

[0135] In one embodiment, the predicted value includes a disease classification predicted value and a first severity classification predicted value. In S5, the disease type and severity of the leaf to be detected are obtained based on the predicted value of the leaf to be detected, specifically including:

[0136] S51, obtaining the disease type of the leaf to be detected according to the disease classification prediction value of the leaf to be detected;

[0137] S52: Obtain the severity of the leaf to be detected according to the first severity classification prediction value of the leaf to be detected.

[0138] Specifically, the leaf disease image to be detected is input into the updated TSTC network, and after processing, the predicted value of the leaf to be detected is output. The predicted value includes: disease classification prediction value, first severity classification prediction value, first overall classification prediction value, second overall classification prediction value and second severity classification prediction value.

[0139] The disease type and severity of the leaf to be tested are predicted based on the disease classification prediction value and the first severity classification prediction value among these prediction values, as follows:

[0140] 1) Obtaining the disease type of the leaf to be detected according to the disease classification prediction value of the leaf to be detected;

[0141] 2) Obtaining the severity of the leaf to be detected according to the first severity classification prediction value of the leaf to be detected.

[0142] The above-mentioned method for identifying vegetation leaf diseases and their severity uses an end-to-end trainable TSTC network, which can autonomously extract required features through learning during training. A three-branch Swin Transformer network is proposed in the multi-task feature extraction module of the TSTC network, which not only realizes multi-task classification but also effectively captures global information and has a wide receptive field. A compact bilinear pooling technique is used to fuse features, making the features more discriminative. A deep supervision module is designed to enhance the feature extraction capability of the hidden layer, thereby improving the TSTC network's recognition performance for vegetation diseases and their severity.

[0143] The above is a detailed introduction to a method for identifying vegetation leaf diseases and their severity provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for identifying vegetation leaf diseases and their severity, characterized in that: The method comprises: S1. Build a TSTC network, which includes a multi-task feature extraction module, a feature fusion module, and a deep supervision module connected in sequence; S2, presetting leaf classification labels and a training set, inputting the leaf disease images in the training set into the multi-task feature extraction module of the TSTC network, and obtaining three sets of shallow features after processing; S3, the feature fusion module of the TSTC network uses a compact bilinear pooling technique to fuse the three groups of shallow features to obtain two groups of compact features; S4, inputting the three groups of shallow features and the two groups of compact features into the deep supervision module of the TSTC network, obtaining corresponding prediction values ​​after processing, calculating the total loss of the TSTC network according to the prediction values ​​and the leaf classification labels, and updating the network parameters of the network through backpropagation to obtain an updated TSTC network; S5. Inputting the diseased image of the leaf to be detected into the updated TSTC network, outputting a predicted value of the leaf to be detected after processing, and obtaining the type and severity of the disease of the leaf to be detected based on the predicted value of the leaf to be detected; The multi-task feature extraction module in S1 includes a blocking module and several task feature extraction sub-modules. Each task feature extraction sub-module includes a linear embedding module or a blocking merging module, and several Swin Transformer block modules. The linear embedding module or the blocking merging module is connected to several Swin Transformer block modules. The blocking module is used to block the input leaf disease image. The several task feature extraction sub-modules are divided into four stages and three branch networks to extract shallow features of the leaf disease image.

2. The method for identifying vegetation leaf diseases and their severity according to claim 1, wherein: S2 inputs the leaves in the training set into the multi-task feature extraction module of the TSTC network, and obtains three sets of shallow features after processing, specifically including: S21, inputting the leaf disease images in the training set into the segmentation module in the multi-task feature extraction module, and outputting a plurality of segmented leaf disease images after processing; S22, extracting shallow shared features of the plurality of segmented leaf disease images using the first-stage network and the second-stage network of the multi-task feature extraction module; S23, using the third-stage network and the first branch in the fourth-stage network of the multi-task feature extraction module to identify the shallow shared features, and output a first shallow feature f1 for the disease type; S24, using the third-stage network of the multi-task feature extraction module and the second branch in the fourth-stage network to identify the shallow shared features, and output a second shallow feature f2 for disease severity; S25. Use the third branch in the third-stage network and the fourth-stage network of the multi-task feature extraction module to identify the shallow shared features, and output the third shallow feature f3 for supervising the TSTC network.

3. The method for identifying vegetation leaf diseases and their severity according to claim 2, wherein: The feature fusion module in S1 includes a first CBP layer and a second CBP layer connected in parallel. The feature fusion module in S3 uses a compact bilinear pooling technique to fuse the three sets of shallow features to obtain two sets of compact features, specifically including: S31, input the first shallow feature f1 and the third shallow feature f3 into the first CBP layer, and fuse them using compact bilinear pooling technology to generate a first compact feature f 13 ; S32: Input the second shallow feature f2 and the third shallow feature f3 into the second CBP layer, and fuse them using the compact bilinear pooling technique to generate the second compact feature f 23 .

4. The method for identifying vegetation leaf diseases and their severity according to claim 3, wherein: The first compact feature and the second compact feature can be expressed as: Where B(·) represents bilinear features, C(·) represents compact features, φ(·) represents low-dimensional projection function, S, U, and K represent positions, f1, f2, and f3 represent the first, second, and third shallow features, respectively, and f13 and f23 are the first and second compact features, respectively.

5. The method for identifying vegetation leaf diseases and their severity according to claim 4, wherein: The deep supervision module in S1 includes several Flatten layers and fully connected layers (FC). S4 inputs the three sets of shallow features and the two sets of compact features into the deep supervision module, and outputs corresponding prediction values ​​after processing, specifically including: S41: Input the first shallow feature f1 to the first Flatten layer and the first fully connected layer connected in sequence, and output the disease classification prediction value after processing. S42, the second compact feature f 23 The input is sent to the second and third fully connected layers connected in sequence, and the first severity classification prediction value is output after processing. S43, the first compact feature f 13 Input to the fourth and fifth fully connected layers connected in sequence, and after processing, output the first overall classification prediction value S44: Input the second shallow feature f2 to the second Flatten layer and the sixth fully connected layer connected in sequence, and output the second overall classification prediction value after processing. S45: Input the third shallow feature f3 into the third Flatten layer and the seventh fully connected layer connected in sequence, and output the second severity classification prediction value after processing.

6. The method for identifying vegetation leaf diseases and their severity according to claim 5, wherein: The total loss of the TSTC network is calculated according to the predicted value and the leaf classification label in S4, specifically including: S46, normalizing the classification prediction value using a softmax classifier to obtain a normalized classification prediction value of the leaf image; S47, calculating the leaf classification loss according to the normalized classification prediction value and the leaf classification label; S48. Calculating an intermediate cross entropy loss and an output cross entropy loss according to the leaf classification loss; S49. Calculate the total loss of the TSTC network according to the sum of the intermediate cross entropy loss and the output cross entropy loss.

7. The method for identifying vegetation leaf diseases and their severity according to claim 6, wherein: The S47 calculates the leaf classification loss according to the normalized classification prediction value and the leaf classification label. The specific formula is: Where, l i is the i-th classification loss of leaf, is the normalized predicted value p of the i-th leaf i The mth component in , is the leaf classification label corresponding to the mth component of the i-th classification, and n is the total number of components in the predicted value.

8. The method for identifying vegetation leaf diseases and their severity according to claim 7, wherein: The calculation formula of the intermediate cross entropy loss in S48 is specifically: The calculation formula of the output cross entropy loss in S48 is specifically: Where, L s is the intermediate cross entropy loss, l3 is the first overall classification loss, l4 is the second overall classification loss, l5 is the second severity classification, L o is the output cross entropy loss, l1 is the disease classification loss, l2 is the first severity classification loss, Represents the weight corresponding to the i-th classification loss (i=1,2,3,4,5).

9. The method for identifying vegetation leaf diseases and their severity according to claim 8, wherein: The predicted value includes a disease classification predicted value and a first severity classification predicted value. The step S5 of obtaining the disease type and severity of the leaf to be detected based on the predicted value of the leaf to be detected specifically includes: S51, obtaining the disease type of the leaf to be detected according to the disease classification prediction value of the leaf to be detected; S52: Obtain the severity of the leaf to be detected according to the first severity classification prediction value of the leaf to be detected.

Images