Crop disease image classification method based on multilayer information fusion

Abstract

The invention relates to a crop disease image classification method based on multilayer information fusion. Compared with the prior art, the defect that a disease area is difficult to accurately position is overcome. The method comprises the following steps: obtaining and preprocessing a crop disease image; constructing a crop disease image classification model; training a crop disease image classification model; obtaining crop disease images to be classified; and obtaining a crop disease image classification result. According to the fusion method, background interference is reduced, and a target tiny disease area is well concerned, so that the identification effectiveness of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of disease image recognition, and more specifically, to a method for classifying crop disease images based on multi-layer information fusion. Background Art

[0002] With the continuous progress of machine vision technology, a series of traditional machine learning methods such as decision tree (DT), random forest (RF), support vector machine (SVM), K-means, Gaussian framework, and naive Bayes have been widely applied in the agricultural field. These technologies are used in crop type classification, disease recognition, disease severity assessment, and crop yield prediction. To a certain extent, this has improved the efficiency and accuracy of disease recognition and provided strong support for agricultural production. However, crop disease images obtained in the natural environment often contain a large amount of redundant information, such as weeds, sky, noise, dead leaves, stems, and bare land. These factors pose a great challenge to disease recognition. Traditional computer vision methods are unable to handle these redundant information effectively and require a large amount of manual preprocessing work. In addition, for different types of crop diseases, specific features need to be selected for recognition, which not only limits the generality of the method but also may lead to insufficient features, thus affecting the recognition accuracy.

[0003] In recent years, the field of deep learning has achieved remarkable rapid development. From the initial deep feedforward neural network to the currently most popular Transformer model. In the application of computer vision for crop diseases, deep learning methods mainly involve three major tasks: crop disease image classification, target disease detection, and target disease image segmentation. These tasks have achieved remarkable results and shown broad application prospects. With the upgrade of intelligent devices, it has become feasible to obtain a large amount of high-definition crop disease image data, which lays a foundation for crop disease recognition. By using these data to train the weights of the deep model, an optimized model can be obtained, which can then be used for real-time crop disease recognition in the actual environment. The Transformer model can well focus on the global information in the image through the self-attention mechanism and has great advantages in capturing context information. However, its complexity and high computational cost and number of parameters have become obstacles to its application on limited intelligent devices. In addition, the Transformer model also has deficiencies in capturing local detail information of images. This means that when extracting some small crop disease features, the performance of the Transformer model may not be ideal.

[0004] To improve the effectiveness of the model in identifying crop diseases, deep learning is designed to solve the problem of identifying crop disease images in real environments at the cost of increased code complexity. In current crop disease recognition models, a common practice is to directly classify and identify the features extracted from the last layer of the neural network. However, there may be a potential problem with this approach, that is, the extracted features may be relatively limited and unable to fully express the shallow semantic information in the image. And this shallow semantic information has good local detail information, which is often crucial for accurately identifying diseases.

[0005] Therefore, in order to more effectively identify crop diseases, it is necessary to find a model or method that can extract both global information and retain and enhance shallow semantic information and local detail information. Such a method will help improve the accuracy and efficiency of disease recognition, especially on resource-constrained intelligent devices. Additionally, considering that currently, due to geographical environmental factors, crop diseases photographed in different scenarios make it difficult for deep learning methods to effectively focus on the target diseases, thus reducing the recognition effectiveness. On crop leaves, diseases are also prone to occur densely and in multiple types, which poses challenges for the deep model to accurately identify the disease area. That is, in the existing technology, the deep recognition model only focuses on the features of the last layer and ignores the local detail information extracted by the shallow network, resulting in insufficient information utilization. In addition, the target sizes and shapes of crop disease images in complex environments vary, and the traditional single-scale processing method is difficult to comprehensively capture their detail information, making it difficult for the model to accurately locate the disease area. Summary of the Invention

[0006] The object of the present invention is to solve the defect in the prior art that it is difficult to accurately locate the disease area. Therefore, a crop disease image classification method based on multi-layer information fusion is proposed to solve the above problems.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] A crop disease image classification method based on multi-layer information fusion includes the following steps:

[0009] Obtaining and preprocessing crop disease images: Obtaining crop disease data images and performing preprocessing;

[0010] Constructing a crop disease image classification model: Constructing a crop disease image classification model based on SE-DenseNet, a multi-layer information feature fusion module, and a GRU location information fusion classifier;

[0011] Training the crop disease image classification model: Training the crop disease image classification model using the preprocessed crop disease data images;

[0012] Obtaining the crop disease image to be classified: Obtain the crop disease image to be classified and perform preprocessing;

[0013] Obtaining the classification result of the crop disease image: Input the preprocessed crop disease image to be classified into the trained crop disease image classification model to obtain the classification result of the crop disease image.

[0014] Constructing the crop disease image classification model includes the following steps:

[0015] Set the crop disease image classification model to include SE-DenseNet, a multi-layer information feature fusion module, and a GRU location information fusion classifier;

[0016] Set the input of SE-DenseNet to be an image of 224×224×3. Perform extraction and multi-scale feature information processing and fusion through SE-DenseNet. Layer1 outputs feature information with a dimension of 28×28×96, Layer2 outputs feature information with a dimension of 14×14×112, Layer3 outputs feature information with a dimension of 7×7×120, and Layer4 outputs feature information with a dimension of 7×7×248. The input of the multi-layer information feature fusion module is the features extracted by Layer1, Layer2, Layer3, and Layer4 of SE-DenseNet, and the output is feature information with a dimension of 28×28×96. The input of the GRU location information fusion classifier is the 28×28×96-dimensional feature obtained by the multi-layer information feature fusion module and the 1×248-dimensional feature vector obtained by global average pooling of the output of the last layer of SE-DenseNet (7×7×248), and the output is 12 types of crop disease types;

[0017] Set SE-DenseNet:

[0018] Set SE-DenseNet to include four SE-Dense modules. Transition layers are connected between the four SE-Dense modules. The transition layer consists of a 3×3 convolution with a stride of 2, which is used to compress the number of channels and reduce the size of the feature map;

[0019] Set the SE-Dense module to include four layers. Each layer consists of a 1×1 convolution, a 3×3 convolution, and an SE attention mechanism. The 1×1 convolution is used to compress the number of channels and reduce the computational amount. The 3×3 convolution is used to extract disease features. The SE attention mechanism is used to establish the connection between channels to improve the reusability of channels. The connection between layers in a dense manner improves the model's ability to express the scale of disease features;

[0020] The multi-layer information feature fusion module is set to include four 1×1 convolutions and three selective multi-scale feature fusion mechanisms, and the selective multi-scale feature fusion mechanisms are set;

[0021] The multi-layer information feature fusion module takes the output features of Layer1, Layer2, Layer3, and Layer4 of SE-DenseNet as inputs. First, 1×1 convolutions are used to compress the feature channels of each layer and filter out useless feature information to obtain feature information with 96 channels. Then, the compressed features of Layer3 and Layer4 are input into the first selective multi-scale feature fusion mechanism to obtain a fused feature of size 7×7×96. Subsequently, the fused feature of 7×7×96 and the compressed feature of Layer2 are input into the second selective multi-scale feature fusion mechanism to obtain a fused feature of size 14×14×96. Finally, the fused feature of 14×14×96 and the compressed feature of Layer1 are input into the third selective multi-scale feature fusion mechanism to obtain the final fused feature of size 28×28×96;

[0022] Set the GRU location information fusion classifier: The GRU location information fusion classifier consists of a 1×1 convolutional layer, two GRU recurrent neural networks, and a fully connected layer classifier;

[0023] The GRU location information fusion classifier compresses the features obtained by the multi-layer information feature fusion module into single-channel spatial information through the 1×1 convolutional layer; the single-channel spatial information features are flattened through the Flatten operation to obtain a one-dimensional vector. For the target regions with the same disease, the one-dimensional vector values have the same characteristics, and the targets with the same disease characteristics or multiple regions with the same disease are represented by target 1, target 2,..., target n; A GRU recurrent neural network is used to compress the one-dimensional information and memorize the neuron characteristics of the same disease. The input hidden state h 0 of this GRU recurrent neural network is initialized to 0, and the compressed hidden state h 1 is equal to the number of channels of the last layer of features extracted by the SE-DenseNet network; h 1 and the one-dimensional feature vector obtained by the SE-DenseNet network are used as the hidden state and input value of the second GRU recurrent neural network respectively to obtain a fused output vector, and the result is input into the fully connected layer classifier to obtain the final class output.

[0024] The training of the crop disease image classification model includes the following steps:

[0025] Training of SE-DenseNet:

[0026] The crop disease images Input the SE-DenseNet network to obtain the outputs of Layer1, Layer2, Layer3, and Layer4;

[0027] Calculate the loss value L using the following formula f and optimize the network parameters,

[0028]

[0029] where f represents the recognition network SE-DenseNet, y i represents the true label of the i-th data, x i ' represents the i-th input data in x', and L f represents the loss value calculated by the SE-DenseNet model, and B represents the number of crop disease images input in each batch;

[0030] Training of the multi-layer information feature fusion module: The multi-layer information feature fusion module performs information processing and fusion on the multi-scale feature information of each layer of SE-DenseNet; The 7×7×248-dimensional feature information extracted by Layer4 and the 7×7×120-dimensional feature information extracted by Layer3 are subjected to feature selection and matching through a 1×1 convolutional kernel to make their feature channel number dimensions the same, and the compressed feature information is input into the selective multi-scale feature fusion mechanism for information fusion; The fused feature information is successively fused with the feature information output by Layer2 and Layer1 to obtain the finally output 28×28×96-dimensional feature information;

[0031] The four-layer output features of the SE-DenseNet network are fused through the multi-layer information feature fusion module to obtain fused features, and then the fused features and the results of Layer4 are input into the GRU location information fusion classifier to obtain the class output. Finally, the weight parameters of the multi-layer information feature fusion module and the GRU location information fusion classifier are fine-tuned through the cross-entropy loss function. The calculation formula for the fine-tuning process is as follows:

[0032]

[0033] where π represents the overall model, y i represents the true label of the i-th data, x i represents the i-th input data in x, and L π represents the loss value calculated by the overall model, and B represents the number of crop disease images input in each batch;

[0034] Training of the GRU location information fusion classifier:

[0035] The fused features are compressed into 28×28×1 dimensional information through a 1×1 convolutional kernel to highlight the location information of multi-target diseases in each image; the fused features are flattened to obtain a 1×784 one-dimensional vector, which is input together with the 1×248 dimensional vector obtained by global average pooling GAP of the features of the last layer of SE-DenseNet into two GRU recurrent neural networks and a fully-connected layer classifier to obtain the output of crop disease categories; the first GRU is used to memorize and compress the location information extracted by the fusion module, and the second GRU is used to fuse the location information and the feature information extracted by SE-DenseNet; finally, the vector obtained by the second GRU is input into the fully-connected layer classifier to obtain the category output.

[0036] The training of the SE-DenseNet includes the following steps:

[0037] For crop disease images of size 224×224×3, a feature map of size 56×56×24 is obtained through max-pooling operations with convolutional kernel sizes of 7×7 and 3×3.

[0038] Input the first layer of SE-Dense module and transition layer Layer1 to obtain features of size 28×28×96.

[0039] Input the second layer of SE-Dense module and transition layer Layer2 to obtain features of size 14×14×112.

[0040] Input the third layer of SE-Dense module and transition layer Layer3 to obtain features of size 7×7×120.

[0041] Input the fourth layer of SE-Dense module Layer4 to obtain features of size 7×7×248, and through the global average pooling layer, obtain a 1×248 dimensional feature vector.

[0042] The setting of the selective multi-scale feature fusion mechanism includes the following steps:

[0043] Use the deep features as the weights of the attention mechanism to filter out the location information features related to crop disease recognition in the shallow features.

[0044] Adopt the EMA attention mechanism with cross-space learning ability for the deep features and the shallow features Since the scale size of the deep features is smaller than that of the shallow features, first perform bilinear interpolation on f high to transform it into an output feature map with the same dimension as f low The calculation formula is as follows: The calculation formula is as follows:

[0045] f high_out f(x,y) = (1 - dx)(1 - dy)f high (x1,y1) + dx(1 - dy)f high (x2,y1) + (1 - dx)dyf high (x1,y2) + dxdyf high (x2,y2),

[0046] Among them, (x1,y1), (x2,y2), (x1,y2) and (x2,y1) represent the coordinates of the four pixels closest to the new pixel coordinates, dx and dy represent the differences between the new pixel coordinates and the closest pixel coordinates, and f high is the original image pixel, and f high_out is the interpolated image pixel;

[0047] Take as the input of the EMA attention mechanism, use the groups operation to reshape some channels to the batch dimension, and group the channel dimension into C / / r sub - features, denoted as

[0048] Subsequently, use average pooling kernels of size (H,1) and (1,W) to calculate each grouped sub - feature along the horizontal coordinate direction and the vertical coordinate direction. The output formula for the c - th channel with height H is:

[0049]

[0050] where x c is the c - th input feature map of f h ′ igh_out , W is the width of the feature map, is the output result of the c - th feature map in the height direction of the feature map; Similarly, the output formula for the c - th channel with width W is:

[0051]

[0052] where x c is the c - th input feature map of f h ′ igh_out , H is the height of the feature map, is the output result of the c - th feature map in the width direction of the feature map;

[0053] Connect the two - direction features of z H and z W through Concat, and use a 1×1 - sized convolutional kernel with shared weights for transformation. The formula is as follows:

[0054] ρ = σ(P 1 ([zH , z W ))

[0055] Among them, P 1 represents a 1×1 point convolution, represents the intermediate feature map of the spatial information encoded in the horizontal and vertical directions, r represents the compression ratio factor, and σ represents the ReLU non-linear activation function;

[0056] Slice ρ along the spatial dimension into two separate tensors and Then use the sigmoid function to obtain the weight values in the horizontal and vertical directions. The formula is as follows:

[0057]

[0058] Among them, δ represents the sigmoid activation function, and the outputs and are the attention weights in the H direction and the W direction;

[0059] Multiply the feature map at the corresponding position by the attention weights in the two directions to obtain the 1×1 branch attention feature map. The calculation formula is as follows:

[0060]

[0061] Among them, (i, j) is the coordinate of the i-th row and the j-th column, x t represents the input feature map of the t-th channel, represents the corresponding weight in the t-th channel in the H direction, represents the corresponding weight in the t-th channel in the W direction, y t represents the output feature map of the t-th channel. Therefore, the sub-feature maps weighted by C / / r channels are

[0062] The input f h ′ igh_out will be obtained through a 3×3 convolution operation to expand the receptive field of the feature space information,

[0063] G = σ(P 3 (f′ high_out ))

[0064] Among them, P 3 represents a 3×3 convolution, represents the feature map with a higher receptive field, denoted as the 3×3 branch. Here, r represents the compression ratio factor, and σ represents the ReLU non-linear activation function;

[0065] The feature aggregation of the 1×1 branch attention feature map y and the 3×3 branch feature map G is performed in the Gross-spatial learning manner, and the calculation formula is as follows:

[0066] T = G·θ(GAP(GN(y))) + GN(y)·θ(GAP(G)),

[0067] where θ represents the softmax activation function, GN represents GroupNorm, and GAP represents global average pooling, represents the output information after feature aggregation, y is the 1×1 branch attention feature map, and G is the 3×3 branch feature map;

[0068] The aggregated feature T is passed through the sigmoid function to obtain the spatial attention weight value and is fused with f low and the feature map f high_out for fusion, and the calculation formula is as follows:

[0069] f out = f low × δ(T) + f high_out ,

[0070] where δ represents the sigmoid activation function, represents the fused feature map, f low represents the shallow feature, and f high_out represents the value of the deep feature after bilinear interpolation.

[0071] Beneficial effects

[0072] A method for classifying crop disease images based on multi-layer information fusion according to the present invention proposes a multi-layer information fusion method based on a lightweight dense scale network compared with the prior art to achieve effective identification of crop diseases.

[0073] The method of the present invention extracts crop disease features at multiple scales through a dense network with a Squeeze-and-Excitation attention mechanism (SE-DenseNet), and then fuses the feature information of each layer of the network through a multi-layer pyramid information feature fusion (MPIFF) module to reuse low-level to high-level semantic features and filter out background interference to obtain the position of the effective disease area. Finally, a GRU location fusion classifier (GRU-LIFC) is used to fuse the abstract features extracted by the SE-DenseNet and the target area location information to obtain the classification result. This fusion method reduces background interference and well focuses on the target tiny disease area, thereby improving the recognition effectiveness of the model.

[0074] On the 12-class crop disease dataset constructed in the present invention, the recognition accuracy reaches 98.3%, the number of parameters is only 2.8M, and the computational load is only 1.3G. Compared with the optimal Swin Transformer, the accuracy, precision, and recall are improved by 1.8%, 0.6%, and 6.1% respectively. This method has the characteristics of higher accuracy and fewer parameters, and can be used for effective and high-precision recognition of crop diseases in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is the sequence diagram of the method of the present invention;

[0076] Figure 2 It is the schematic structural diagram of the crop disease image classification model involved in the present invention;

[0077] Figure 3 It is the schematic structural diagram of the SE-Dense module involved in the present invention;

[0078] Figure 4 It is the schematic structural diagram of the multi-layer information feature fusion module involved in the present invention;

[0079] Figure 5 It is the schematic structural diagram of the GRU position information fusion classifier involved in the present invention;

[0080] Figure 6a It is the comparison diagram of the loss value curves of two classifiers for 200 iterations;

[0081] Figure 6b It is the comparison diagram of the accuracy value curves of two classifiers for 200 iterations;

[0082] Figure 7 It is the confusion matrix analysis diagram of 12-class crop disease data;

[0083] Figure 8 It is the Grad-CAM visualization result diagram of crop disease data recognition. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0084] To further understand and recognize the structural features and achieved effects of the present invention, the following is a detailed description with preferred embodiments and accompanying drawings:

[0085] As Figure 1 shown, a method for classifying crop disease images based on multi-layer information fusion according to the present invention includes the following steps:

[0086] The first step is the acquisition and preprocessing of crop disease images: acquiring crop disease data images and performing preprocessing.

[0087] Step 2: Construct a crop disease image classification model: As Figure 2 shown, construct a crop disease image classification model based on SE-DenseNet, a multi-layer information feature fusion module, and a GRU location information fusion classifier.

[0088] SE-DenseNet consists of densely connected SE-Dense modules, which enables the model to have a diversity of receptive fields, thus improving the expression ability for disease scale differences. The multi-layer information feature fusion module can effectively fuse feature maps with different scale sizes in each layer of SE-DenseNet to achieve feature reuse. In addition, taking the finally fused features as the target disease location information can improve the recognition effectiveness of the entire model. The GRU location information fusion classifier promotes the effectiveness and accuracy of the model in disease recognition by combining the features of the last layer of SE-DenseNet with the location information of the target disease area.

[0089] (1) Set that the crop disease image classification model includes SE-DenseNet, a multi-layer information feature fusion module, and a GRU location information fusion classifier;

[0090] Set that the input of SE-DenseNet is an image of 224×224×3, and perform multi-scale feature information extraction and fusion through SE-DenseNet. Layer1 outputs feature information with a dimension of 28×28×96, Layer2 outputs feature information with a dimension of 14×14×112, Layer3 outputs feature information with a dimension of 7×7×120, and Layer4 outputs feature information with a dimension of 7×7×248. The input of the multi-layer information feature fusion module is the features extracted by Layer1, Layer2, Layer3, and Layer4 of SE-DenseNet, and the output is feature information with a dimension of 28×28×96. The input of the GRU location information fusion classifier is the feature information with a dimension of 28×28×96 obtained by the multi-layer information feature fusion module and the 1×248-dimensional feature vector obtained by global average pooling of the output of the last layer of SE-DenseNet with a dimension of 7×7×248, and the output is 12 types of crop disease types.

[0091] (2) Set SE-DenseNet: Set that SE-DenseNet includes four layers of SE-Dense modules, and transition layers are connected between the four layers of SE-Dense modules. The transition layer consists of a 3×3 convolution with a stride of 2, which is used to compress the number of channels and reduce the size of the feature map;

[0092] The set SE-Dense module consists of four layers, and each layer is composed of a 1×1 convolution, a 3×3 convolution, and an SE attention mechanism. The 1×1 convolution is used to compress the number of channels and reduce the computational amount. The 3×3 convolution is used to extract disease features. The SE attention mechanism is used to establish the connection between channels to improve the reusability of channels. The connection between layers in a dense manner improves the model's ability to express the scale of disease features.

[0093] Due to the large differences in the intra-class and inter-class scales of crop diseases collected and obtained in different environments and periods, the present invention designs a lightweight SE-DenseNet mainly composed of dense connection blocks SE-Dense modules for feature extraction. Figure 3 As shown, the SE-Dense module structure includes 4 layers. Each layer is composed of a 1×1 convolution block, a 3×3 convolution block, and an SE attention mechanism, and connects the features extracted by all previous layers in a Concat manner. This dense connection method alleviates the problem of gradient disappearance, enhances feature propagation, promotes feature reuse, and greatly reduces the number of parameters. In addition, different lines with multiple dense connections have different receptive fields, improving the diversity of the model's ability to extract features with different scales. We inserted an SE attention mechanism after each layer of convolution to improve the model's attention to disease features. The SE attention mechanism is a technique that focuses on the importance of each channel in the image feature map.

[0094] In the SE-DenseNet convolutional neural network, although more feature channel information is aggregated through dense connection, each channel of the feature map represents a specific feature information, so the contribution of different channels to the final task often varies. The SE attention mechanism dynamically adjusts the weight of each channel, enabling the network to focus on the features more critical for crop disease recognition, thereby improving the overall performance. By reusing Layer1 to Layer4, our method does not require the subsequent layer to aggregate more channel features relative to the previous layer like existing dense networks, which greatly reduces the number of parameters brought by convolutional calculations.

[0095] (3) The crop disease features extracted by the convolutional neural network transition from low-level to high-level. The features extracted in its shallow layers mainly contain low-level details such as edges, textures, and shapes. As the number of network layers deepens, the receptive field becomes larger and larger, and the feature map output contains more and more abstract semantic information. In addition, in the recognition and classification network, the layer space size gradually decreases, and the number of feature channels gradually increases. To better utilize the discriminant information contained in different network layers, the present invention designs a multi-layer pyramid information feature fusion module (Multi-layer pyramid information feature fusionmodule), such as Figure 4As shown in the figure. For the features obtained from different network layers, first, 1×1 convolutional kernels are used to screen the feature maps of different scales and match the feature channel dimensions to be consistent. Then, the semantic feature information of different scales is fused in sequence from high-level to low-level in a bottom-up order. The present invention adopts a Selective Multi-Scale Feature Fusion (SMFF) mechanism to combine the screened low-level features and high-level features. After the high-level features are expanded, scale adjustment is performed through bilinear interpolation, and then they are fused with the low-level features, thereby enhancing the model's ability to express crop disease features.

[0096] It is set that the multi-layer information feature fusion module includes four 1×1 convolutions and three selective multi-scale feature fusion mechanisms, and the selective multi-scale feature fusion mechanism is set.

[0097] Setting the selective multi-scale feature fusion mechanism includes the following steps:

[0098] A1) Using the deep features as the weights of the attention mechanism, screening out the position information features related to crop disease recognition in the shallow features;

[0099] Adopting the EMA attention mechanism with cross-space learning ability, for the deep features and the shallow features Since the scale of the deep features is smaller than that of the shallow features, first, f is transformed into an output feature map with the same dimension as f through bilinear interpolation high low The calculation formula is as follows:

[0100] f high_out (x,y)=(1 - dx)(1 - dy)f high (x1,y1)+dx(1 - dy)f high (x2,y1)+(1 - dx)dyf high (x1,y2)+dxdyf high (x2,y2),

[0101] where, (x1,y1), (x2,y2), (x1,y2) and (x2,y1) represent the coordinates of the four pixels closest to the new pixel coordinates, dx and dy represent the differences between the new pixel coordinates and the closest pixel coordinates, f high is the original image pixel, and f high_out is the interpolated image pixel;

[0102] A2) Taking as the input of the EMA attention mechanism, using the groups operation to reshape some channels to the batch dimension, and grouping the channel dimension into C / / r sub-features, denoted as

[0103] Subsequently, average pooling kernels of size (H, 1) and (1, W) are used to calculate each sub-feature after grouping along the horizontal and vertical coordinate directions. The output formula for the c-th channel with height H is as follows:

[0104]

[0105] where x c is the c-th input feature map of f h ′ igh_out , W is the width of the feature map, is the output result of the c-th feature map in the height direction of the feature map; Similarly, the output formula for the c-th channel with width W is:

[0106]

[0107] where x c is the c-th input feature map of f h ′ igh_out , H is the height of the feature map, is the output result of the c-th feature map in the width direction of the feature map;

[0108] A3) Connect the two-direction features of z H and z W through Concat and perform transformation using a 1×1 convolutional kernel with shared weights. The formula is as follows:

[0109] ρ = σ(P 1 ([z H , z W )),

[0110] where P 1 represents a 1×1 point convolution, represents the intermediate feature map encoding spatial information in the horizontal and vertical directions, r represents the compression ratio factor, and σ represents the ReLU non-linear activation function;

[0111] A4) Cut ρ into two separate tensors along the spatial dimension and Then use the sigmoid function to obtain the weight values in the horizontal and vertical directions. The formula is as follows:

[0112]

[0113] where δ represents the sigmoid activation function, and the outputs and are the attention weights in the H direction and the W direction;

[0114] A5) Multiply the feature map at the corresponding position by the attention weights in two directions to obtain the 1×1 branch attention feature map. The calculation formula is as follows:

[0115]

[0116] where (i,j) represents the coordinates of the i-th row and the j-th column, and x t represents the input feature map of the t-th channel, represents the weight corresponding to the t-th channel in the H direction, represents the weight corresponding to the t-th channel in the W direction, and y t represents the output feature map of the t-th channel. Therefore, the weighted sub-feature map of C / / r channels is

[0117] A6) The input f h ′ igh_out will be obtained through a 3×3 convolution operation to expand the receptive field of the feature space information,

[0118] G = σ(P 3 (f′ high_out ))

[0119] where P 3 represents the 3×3 convolution, represents the feature map with a higher receptive field, denoted as the 3×3 branch. Here, r represents the compression ratio factor, and σ represents the ReLU non-linear activation function;

[0120] A7) Perform feature aggregation on the 1×1 branch attention feature map y and the 3×3 branch feature map G in the Gross-spatial learning manner. The calculation formula is as follows:

[0121] T = G·θ(GAP(GN(y))) + GN(y)·θ(GAP(G)),

[0122] where θ represents the softmax activation function, GN represents GroupNorm, and GAP represents global average pooling, represents the output information after feature aggregation, y is the 1×1 branch attention feature map, and G is the 3×3 branch feature map;

[0123] A8) After passing the aggregated feature T through the sigmoid function, obtain the spatial attention weight value and fuse it with f low and the feature map f high_out The calculation formula is as follows:

[0124] f out = f low ×δ(T) + f high_out ​

[0125] where δ represents the sigmoid activation function, represents the fused feature map, f low represents the shallow feature, f high_out represents the value of the deep feature after bilinear interpolation.

[0126] The multi-layer information feature fusion module takes the output features of Layer1, Layer2, Layer3, and Layer4 of SE-DenseNet as inputs. First, it uses 1×1 convolution to compress the feature channels of each layer and filter out useless feature information to obtain feature information with 96 channels. Then, it inputs the compressed features of Layer3 and Layer4 into the first selective multi-scale feature fusion mechanism to obtain a fused feature of size 7×7×96. Subsequently, it inputs the fused feature of 7×7×96 and the compressed feature of Layer2 into the second selective multi-scale feature fusion mechanism to obtain a fused feature of size 14×14×96. Finally, it inputs the fused feature of 14×14×96 and the compressed feature of Layer1 into the third selective multi-scale feature fusion mechanism to obtain the final fused feature of size 28×28×96.

[0127] (4) Set the GRU location information fusion classifier: The GRU location information fusion classifier consists of a 1×1 convolutional layer, two GRU recurrent neural networks, and a fully connected layer classifier.

[0128] The deep recognition network often uses global average pooling in the last layer to obtain feature vectors for final classification, but this easily destroys the spatial location information of the discriminant region. Especially for a crop disease image with multiple target disease regions, capturing this location information can improve feature diversity. Therefore, the present invention designs a GRU location information fusion classifier to better fuse the location information of the target disease, thereby improving the image classification and recognition performance, as Figure 5 shown.

[0129] The GRU location information fusion classifier compresses the features obtained by the multi-layer information feature fusion module into single-channel spatial information through a 1×1 convolutional layer; flattens the single-channel spatial information features through the Flatten operation to obtain a one-dimensional vector. For the target regions of the same disease, the one-dimensional vector values have the same characteristics, and the targets with the same disease characteristics or multiple same-disease regions are represented by target 1, target 2,..., target n; uses a GRU recurrent neural network to compress the one-dimensional information and memorize the neuron characteristics of the same disease. The input hidden state h of this GRU recurrent neural network 0is initialized to 0, and the compressed hidden state h 1 is equal to the number of feature channels in the last layer extracted by the SE-DenseNet network; h 1 and the one-dimensional feature vector obtained from the SE-DenseNet network are used as the hidden state and input value of the second GRU recurrent neural network respectively to obtain the fused output vector, and the result is input into the fully connected layer classifier to obtain the final class output.

[0130] Step 3, training of the crop disease image classification model: Use the preprocessed crop disease data images to train the crop disease image classification model. This patent adopts a two-stage training strategy. In the first stage, SE-DenseNet is trained, and in the second stage, the multi-layer information fusion module and the GRU position information fusion classifier are trained. This two-stage training strategy can reduce the consumption of computing resources, energy consumption and training costs of computer devices. This strategy not only improves the performance of the model, but also optimizes the training process, making the whole system more efficient and energy-saving.

[0131] (1) Training of SE-DenseNet:

[0132] Input the crop disease image into the SE-DenseNet network to obtain the outputs of Layer1, Layer2, Layer3, and Layer4.

[0133] B1) For crop disease images of size 224×224×3, since the images contain more redundant information, downsampling is performed through max-pooling with convolutional kernel sizes of 7×7 and 3×3 to obtain a feature map of size 56×56×24. This process helps to reduce the redundant information in the images;

[0134] B2) Input the first SE-Dense module to aggregate multi-scale features and compress useless channel information through the transition layer (Layer1) to obtain a feature of size 28×28×96;

[0135] B3) Input the second SE-Dense module to aggregate multi-scale features and compress useless channel information through the transition layer (Layer2) to obtain a feature of size 14×14×112;

[0136] B4) Input the third SE-Dense module to aggregate multi-scale features and compress useless channel information through the transition layer (Layer3) to obtain a feature of size 7×7×120;

[0137] B5) Input the fourth - layer SE - Dense module to aggregate multi - scale features (Layer4), obtain features of size 7×7×248, and through the global average pooling layer, obtain a 1×248 - dimensional feature vector.

[0138] (2) Calculate the loss value L using the following formula f and optimize the network parameters.

[0139]

[0140] where f represents the recognition network SE - DenseNet, y i represents the true label of the i - th data, x i ′ represents the i - th input data in x′, L f represents the loss value calculated by the SE - DenseNet model, and B represents the number of crop disease images input in each batch.

[0141] (3) Training of the multi - layer information feature fusion module: The multi - layer information feature fusion module processes and fuses the multi - scale feature information of each layer of SE - DenseNet; The 7×7×248 - dimensional feature information extracted from Layer4 and the 7×7×120 - dimensional feature information extracted from Layer3 are subjected to feature selection and matching through a 1×1 - sized convolutional kernel to make their feature channel number dimensions the same, and the compressed feature information is input into the selective multi - scale feature fusion mechanism for information fusion; The fused feature information is successively fused with the feature information output by Layer2 and Layer1, thereby obtaining the finally output 28×28×96 - dimensional feature information;

[0142] The four - layer output features of the SE - DenseNet network are passed through the multi - layer information feature fusion module to obtain fused features, and then the fused features and the results of Layer4 are input into the GRU location information fusion classifier to obtain class outputs. Finally, the weight parameters of the multi - layer information feature fusion module and the GRU location information fusion classifier are fine - tuned through the cross - entropy loss function. The calculation formula for the fine - tuning process is as follows:

[0143]

[0144] where π represents the overall model, y i represents the true label of the i - th data, x i represents the i - th input data in x, L π represents the loss value calculated by the overall model, and B represents the number of crop disease images input in each batch.

[0145] (4) Training of the GRU location information fusion classifier:

[0146] The fused features are compressed into 28×28×1 dimensional information through a 1×1 convolutional kernel to highlight the location information of multi-target diseases in each image; the fused features are flattened to obtain a one-dimensional vector of 1×784, which is input into two GRU recurrent neural networks and a fully-connected layer classifier together with the 1×248 dimensional vector obtained by global average pooling (GAP) of the features of the last layer of SE-DenseNet to obtain the output of the crop disease category; the first GRU is used to memorize and compress the location information extracted by the fusion module, and the second GRU is used to fuse the location information and the feature information extracted by SE-DenseNet; finally, the vector obtained by the second GRU is input into the fully-connected layer classifier to obtain the category output.

[0147] Step 4, acquisition of the crop disease image to be classified: Acquire the crop disease image to be classified and perform preprocessing.

[0148] Step 5, obtaining the classification result of the crop disease image: Input the preprocessed crop disease image to be classified into the trained crop disease image classification model to obtain the classification result of the crop disease image.

[0149] To show that the fusion classifier constructed using GRU is superior to the most commonly used fully-connected layer classifier method currently, an experimental comparison of the training process was carried out on the crop disease data under 12 constructed natural scenes. As Figure 6a and Figure 6b shown, although the classifier constructed using the fully-connected layer and the classifier proposed in this patent achieved similar training losses and changes in training accuracy, the method of the present invention has more stable test losses and test accuracy on the test set, and the test accuracy is higher. This benefits from the memory function of the GRU recurrent neural network, which has better ability to capture long-term dependence relationships.

[0150] Regarding the training strategy adopted in the present invention, the confusion matrix results of the recognition of 12 types of crop disease data were plotted on the test data set, as Figure 7 shown. The method of the present invention achieved good recognition performance on the crop disease data obtained under 12 natural environments. Especially for the recognition accuracy of Rfs, Rbs and Rsb diseases reached 100%. These types of diseases show a dense distribution on the leaves, and the method of the present invention can well capture the global information of the disease location, prompting the model to learn more features, thereby improving the recognition performance.

[0151] To show that the designed multi-layer information fusion module and the GRU location information fusion classifier can improve the model's ability to focus on the location of crop disease target areas in natural scenes, the present invention visualized three common types of crop diseases using Grad-CAM heatmaps, as Figure 8 shown. ByFigure 8 It can be seen that the method of the present invention pays good attention to the crop disease areas. In addition, for smaller crop disease areas, the method of the present invention can also accurately pay attention, which is very important for the effectiveness of identifying crop diseases in the natural environment.

[0152] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification is only the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection required by the present invention is defined by the appended claims and their equivalents.

Claims

1. A crop disease image classification method based on multi-layer information fusion, characterized in that: The following steps are involved: 11) Acquisition and preprocessing of crop disease images: Acquisition of crop disease data images and preprocessing; 12) Construct a crop disease image classification model: Construct a crop disease image classification model based on SE-DenseNet, multi-layer information feature fusion module and GRU position information fusion classifier; 13) Training of crop disease image classification model: Using the preprocessed crop disease data images, the crop disease image classification model is trained; 14) Acquisition of crop disease images to be classified: Acquisition of crop disease images to be classified and preprocessing; 15) Obtaining the classification results of crop disease images: Inputting the preprocessed crop disease images to be classified into the trained crop disease image classification model to obtain the crop disease image classification results.

2. The crop disease image classification method based on multi-layer information fusion according to claim 1 is characterized in that: The construction of the crop disease image classification model comprises the following steps: 21) The crop disease image classification model is set to include SE-DenseNet, multi-layer information feature fusion module and GRU position information fusion classifier; The input of SE-DenseNet is set to be a 224×224×3 image. SE-DenseNet is used to extract multi-scale feature information processing and fusion. Layer1 outputs 28×28×96 dimensional feature information, Layer2 outputs 14×14×112 dimensional feature information, Layer3 outputs 7×7×120 dimensional feature information, and Layer4 outputs 7×7×248 dimensional feature information. The input of the multi-layer information feature fusion module is the features extracted by Layer1, Layer2, Layer3 and Layer4 of SE-DenseNet, and the output is 28×28×96 dimensional features. The input of the GRU position information fusion classifier is the 28×28×96 dimensional features obtained by the multi-layer information feature fusion module and the 7×7×248 output of the last layer of SE-DenseNet. After global average pooling, a 1×248 dimensional feature vector is obtained, and the output is 12 types of crop diseases. 22) Setting up SE-DenseNet: SE-DenseNet is assumed to include four layers of SE-Dense modules, and the four layers of SE-Dense modules are connected with transition layers. The transition layer consists of a 3×3 convolution with a stride of 2, which is used to compress the number of channels and reduce the size of feature maps. The SE-Dense module is set to consist of four layers, each of which consists of 1×1 convolution, 3×3 convolution and SE attention mechanism. 1×1 convolution is used to compress the number of channels and reduce the amount of calculation, 3×3 convolution is used to extract disease features, and SE attention mechanism is used to establish connections between channels to improve channel reusability. Dense connections are used between layers to improve the model's ability to express disease feature scales. 23) setting a multi-layer information feature fusion module including four 1×1 convolutions and three selective multi-scale feature fusion mechanisms, and setting a selective multi-scale feature fusion mechanism; The multi-layer information feature fusion module takes the output features of Layer1, Layer2, Layer3 and Layer4 of SE-DenseNet as input. First, 1×1 convolution is used to compress the feature channels of each layer and filter out useless feature information to obtain feature information with a channel number of 96. Then, the compressed features of Layer3 and Layer4 are input into the first selective multi-scale feature fusion mechanism to obtain a fused feature of size 7×7×96. Subsequently, the fused features of 7×7×96 and the compressed features of Layer2 are input into the second selective multi-scale feature fusion mechanism to obtain a fused feature of size 14×14×96. Finally, the fused features of 14×14×96 and the compressed features of Layer1 are input into the third selective multi-scale feature fusion mechanism to obtain the final fused feature of size 28×28×96. 24) Set the GRU position information fusion classifier: The GRU position information fusion classifier consists of a 1×1 convolutional layer, two GRU recurrent neural networks and a fully connected layer classifier; The GRU position information fusion classifier compresses the features obtained by the multi-layer information feature fusion module into the spatial information of a single channel through a 1×1 convolutional layer; the spatial information features of a single channel are flattened through the Flatten operation to obtain a one-dimensional vector. For the target area with the same disease, its one-dimensional vector value has the same characteristics, and target 1, target 2, ..., target n are used to represent targets with the same disease characteristics or multiple same disease areas; a GRU recurrent neural network is used to compress the one-dimensional information and memorize the neuronal characteristics of the same disease. The input hidden state h0 of the GRU recurrent neural network is initialized to 0, and the compressed hidden state h1 is equal to the number of feature channels of the last layer extracted by the SE-DenseNet network; h1 and the one-dimensional feature vector obtained by the SE-DenseNet network are used as the hidden state and input value of the second GRU recurrent neural network respectively to obtain the fused output vector, and the result is input into the fully connected layer classifier to obtain the final category output.

3. The crop disease image classification method based on multi-layer information fusion according to claim 1 is characterized in that: The training of the crop disease image classification model includes the following steps: 31) SE-DenseNet training: Crop disease images Input the SE-DenseNet network and get the output of Layer1, Layer2, Layer3, and Layer4; 32) Use the following formula to calculate the loss value L f And optimize network parameters, Among them, f represents the recognition network SE-DenseNet, y i represents the true label of the i-th data, x i ′ represents the i-th input data in x′, L f represents the loss value calculated by the SE-DenseNet model, and B represents the number of crop disease images input in each batch; 33) Training of multi-layer information feature fusion module: The multi-layer information feature fusion module performs information processing and fusion on the multi-scale feature information of each layer of SE-DenseNet; the 7×7×248 dimensional feature information extracted by Layer4 and the 7×7×120 dimensional feature information extracted by Layer3 are selected and matched through a 1×1 convolution kernel to make their feature channel number dimensions the same, and the compressed feature information is input into the selective multi-scale feature fusion mechanism for information fusion; the fused feature information is fused with the feature information output by Layer2 and Layer1 in turn, so as to obtain the final output 28×28×96 dimensional feature information; The four-layer output features of the SE-DenseNet network are fused through the multi-layer information feature fusion module to obtain the fusion features, and then the fusion features and the results of Layer4 are input into the GRU position information fusion classifier to obtain the category output. Finally, the weight parameters of the multi-layer information feature fusion module and the GRU position information fusion classifier are fine-tuned through the cross entropy loss function. The calculation formula used in the fine-tuning process is as follows: Among them, π represents the overall model, y i represents the true label of the i-th data, x i represents the i-th input data in x, L π represents the loss value calculated by the overall model, and B represents the number of crop disease images input in each batch; 34) Training of GRU position information fusion classifier: The fused features are compressed into 28×28×1 dimensional information through a 1×1 convolution kernel to highlight the location information of multiple target diseases in each image; the fused features are flattened to obtain a 1×784 one-dimensional vector, and are input into two GRU recurrent neural networks and a fully connected layer classifier together with a 1×248 dimensional vector obtained by global average pooling GAP of the last layer of SE-DenseNet features to obtain the output of crop disease categories; the first GRU is used to memorize and compress the location information extracted by the fusion module, and the second GRU is used to fuse the location information and the feature information extracted by SE-DenseNet; finally, the vector obtained by the second GRU is input into the fully connected layer classifier to obtain the category output.

4. The crop disease image classification method based on multi-layer information fusion according to claim 3 is characterized in that: The training of SE-DenseNet includes the following steps: 41) For a crop disease image of size 224×224×3, a feature map of size 56×56×24 is obtained through a maximum pooling operation with convolution kernel sizes of 7×7 and 3×3; 42) Input the first layer SE-Dense module and transition layer Layer1 to obtain a feature size of 28×28×96; 43) Input the second layer SE-Dense module and transition layer Layer2 to obtain a feature of size 14×14×112; 44) Input the third layer SE-Dense module and transition layer Layer3 to obtain a feature size of 7×7×120; 45) Input the fourth layer SE-Dense module Layer4 to obtain a feature of size 7×7×248, and pass it through the global average pooling layer to obtain a 1×248 dimensional feature vector.

5. The crop disease image classification method based on multi-layer information fusion according to claim 2 is characterized in that: The setting of the selective multi-scale feature fusion mechanism comprises the following steps: 51) Using deep features as the weight of the attention mechanism, the location information features related to crop disease identification in shallow features are screened out; The EMA attention mechanism with cross-space learning ability is used for deep features. and shallow features Since the scale of deep features is smaller than that of shallow features, we first use bilinear interpolation to convert f high Transformed into and f low Output feature maps with the same dimensions The calculation formula is as follows: f high_out (x,y)=(1-dx)(1-dy)f high (x1,y1)+dx(1-dy)f high (x2,y1)+(1-dx)dyf high (x1,y2)+dxdyf high (x2,y2), Among them, (x1, y1), (x2, y2), (x1, y2) and (x2, y1) represent the coordinates of the four pixels closest to the new pixel coordinates, dx and dy represent the difference between the new pixel coordinates and the nearest pixel coordinates, and f high is the original image pixel, f high_out Interpolated image pixels; 52) As the input of the EMA attention mechanism, the groups operation is used to reshape some channels to the batch dimension, and the channel dimension is grouped into C / / r sub-features, recorded as Then, the average pooling kernels of size (H, 1) and (1, W) are used to calculate each sub-feature after grouping along the horizontal coordinate direction and the vertical coordinate direction. The output formula of the cth channel with a height of H is: Among them, x c f h ' igh_out The cth input feature map of , W is the width of the feature map, is the output result of the cth feature map in the height direction of the feature map; similarly, the output formula of the cth channel with a width of W is: Among them, x c f h ' igh_out The cth input feature map, H is the height of the feature map, is the output result of the cth feature map in the width direction of the feature map; 53) Use Concat to convert z H and z w The two directional features are concatenated and transformed using a 1×1 convolution kernel with shared weights, as follows: ρ=σ(P1([z h ,z W ])), Among them, P1 represents 1×1 point convolution, Represents the intermediate feature map encoding spatial information in the horizontal and vertical directions, r represents the compression scale factor, and σ represents the ReLU nonlinear activation function; 54) Split ρ into two separate tensors along the spatial dimension and Then use the sigmoid function to get the weight values ​​in the horizontal and vertical directions. The formula is as follows: Where δ represents the sigmoid activation function, the output and is the attention weight in the H direction and the W direction; 55) Multiply the feature map of the corresponding position and the attention weights of the two directions to obtain a 1×1 branch attention feature map. The calculation formula is as follows: Where (i, j) is the i-th row, j-th column coordinate, x t represents the t-th channel input feature map, Represents the weight corresponding to the tth channel in the H direction, Represents the weight corresponding to the tth channel in the W direction, y t represents the output feature map of the tth channel, and the weighted sub-feature map of C / / r channels is 56) Input f h ' igh_out The 3×3 convolution operation will be used to obtain Used to expand the receptive field of feature space information, G=σ(P3(f′ high_out )), Where P3 represents 3×3 convolution, Represents a feature map with higher sensitivity, recorded as a 3×3 branch, where r represents the compression scale factor and σ represents the ReLU nonlinear activation function; 57) Perform Gross-spatial learning on the 1×1 branch attention feature map y and the 3×3 branch feature map G to perform feature aggregation. The calculation formula is as follows: T=G·θ(GAP(GN(y)))+GN(y)·θ(GAP(G)), Where θ represents the softmax activation function, GN represents GroupNorm, and GAP represents global average pooling. Represents the output information after feature aggregation, y is a 1×1 branch attention feature map, and G is a 3×3 branch feature map; 58) The aggregated feature T is passed through the sigmoid function to obtain the spatial attention weight value and combined with f low And the feature map f high_out The calculation formula for fusion is as follows: f out =f low ×δ(T)+f high_out , Where δ represents the sigmoid activation function, Represents the fused feature map, f low represents shallow features, f high_out Represents the value of the deep feature after bilinear interpolation.

Images