A Crop Disease Image Classification Method Based on Multi-Layer Information Fusion

Abstract

The present invention relates to a method for classifying crop disease images based on multi-layer information fusion, which solves the defect of difficult to accurately locate the disease area compared with the prior art. The present invention includes the following steps: acquisition and preprocessing of crop disease images; construction of a crop disease image classification model; training of the crop disease image classification model; acquisition of crop disease images to be classified; and obtaining of the classification results of crop disease images. The fusion method of the present invention reduces background interference and well focuses on the target tiny disease area, thereby improving the recognition effectiveness of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of disease image recognition, and more specifically, to a method for classifying crop disease images based on multi-layer information fusion. Background Art

[0002] With the continuous progress of machine vision technology, a series of traditional machine learning methods such as decision tree (DT), random forest (RF), support vector machine (SVM), K-means, Gaussian framework, and naive Bayes have been widely applied in the agricultural field. These technologies are used in crop type classification, disease recognition, disease severity assessment, and crop yield prediction. This has improved the efficiency and accuracy of disease recognition to a certain extent and provided strong support for agricultural production. However, crop disease images obtained in the natural environment often contain a large amount of redundant information, such as weeds, sky, noise, dead leaves, stems, and bare land. These factors pose great challenges to disease recognition. Traditional computer vision methods are unable to handle these redundant information effectively and require a large amount of manual preprocessing work. In addition, for different types of crop diseases, specific features need to be selected for recognition, which not only limits the generality of the method but may also lead to insufficient features, thus affecting the recognition accuracy.

[0003] In recent years, the field of deep learning has achieved remarkable rapid development. From the initial deep feedforward neural network to the currently most popular Transformer model. In the application of computer vision for crop diseases, deep learning methods mainly involve three major tasks: crop disease image classification, target disease detection, and target disease image segmentation. These tasks have achieved remarkable results and shown broad application prospects. With the upgrade of intelligent devices, it has become feasible to obtain a large amount of high-definition crop disease image data, which lays a foundation for crop disease recognition. By using these data to train the weights of the deep model, an optimized model can be obtained, which can be used for real-time crop disease recognition in the actual environment. The Transformer model can well focus on the global information in the image through the self-attention mechanism and has great advantages in capturing context information. However, its complexity and high computational cost and number of parameters have become obstacles to its application on restricted intelligent devices. In addition, the Transformer model also has deficiencies in capturing local detailed information of images. This means that when extracting some tiny crop disease features, the performance of the Transformer model may not be ideal.

[0004] To improve the effectiveness of the model in identifying crop diseases, deep learning is designed to solve the problem of identifying crop disease images in real environments at the cost of increased code complexity. In current crop disease recognition models, a common practice is to directly classify and identify the features extracted from the last layer of the neural network. However, there may be a potential problem with this approach, that is, the extracted features may be relatively limited and unable to fully express the shallow semantic information in the image. And this shallow semantic information has good local detail information, which is often crucial for accurately identifying diseases.

[0005] Therefore, in order to more effectively identify crop diseases, it is necessary to find a model or method that can extract global information, retain and enhance shallow semantic information and local detail information. Such a method will help improve the accuracy and efficiency of disease recognition, especially on resource-constrained intelligent devices. Additionally, considering that currently, due to geographical environmental factors, crop diseases photographed in different scenarios make it difficult for deep learning methods to effectively focus on the target diseases, thus reducing the recognition effectiveness. On crop leaves, diseases are also prone to occur densely and in multiple types, which poses challenges for the deep model to accurately identify the disease area. That is, in the existing technology, the deep recognition model only focuses on the features of the last layer and ignores the local detail information extracted by the shallow network, resulting in insufficient information utilization. In addition, the target sizes and shapes of crop disease images in complex environments are diverse, and traditional single-scale processing methods are difficult to comprehensively capture their detail information, making it difficult for the model to accurately locate the disease area. Summary of the Invention

[0006] The purpose of the present invention is to solve the defect in the prior art that it is difficult to accurately locate the disease area. Therefore, a crop disease image classification method based on multi-layer information fusion is proposed to solve the above problems.

[0007] To achieve the above purpose, the technical solution of the present invention is as follows:

[0008] A crop disease image classification method based on multi-layer information fusion includes the following steps:

[0009] Obtaining and preprocessing crop disease images: Obtaining crop disease data images and performing preprocessing;

[0010] Constructing a crop disease image classification model: Constructing a crop disease image classification model based on SE-DenseNet, a multi-layer information feature fusion module, and a GRU location information fusion classifier;

[0011] Training the crop disease image classification model: Training the crop disease image classification model using the preprocessed crop disease data images;

[0012] Obtaining of crop disease images to be classified: Obtain crop disease images to be classified and perform preprocessing;

[0013] Obtaining of crop disease image classification results: Input the preprocessed crop disease images to be classified into the trained crop disease image classification model to obtain crop disease image classification results.

[0014] The construction of the crop disease image classification model includes the following steps:

[0015] Set the crop disease image classification model to include SE-DenseNet, a multi-layer information feature fusion module, and a GRU location information fusion classifier;

[0016] Set the input of SE-DenseNet to be an image of 224×224×3. Through SE-DenseNet, multi-scale feature information extraction and fusion processing are performed. Layer1 outputs feature information with a dimension of 28×28×96, Layer2 outputs feature information with a dimension of 14×14×112, Layer3 outputs feature information with a dimension of 7×7×120, and Layer4 outputs feature information with a dimension of 7×7×248. The input of the multi-layer information feature fusion module is the features extracted by Layer1, Layer2, Layer3, and Layer4 of SE-DenseNet, and the output is feature information with a dimension of 28×28×96. The input of the GRU location information fusion classifier is the 28×28×96-dimensional feature obtained by the multi-layer information feature fusion module and the 1×248-dimensional feature vector obtained by global average pooling of the output of the last layer of SE-DenseNet (7×7×248), and the output is 12 types of crop disease types;

[0017] Set SE-DenseNet:

[0018] Set SE-DenseNet to include four SE-Dense modules. Transition layers are connected between the four SE-Dense modules. The transition layer consists of a 3×3 convolution with a stride of 2, which is used to compress the number of channels and reduce the size of the feature map;

[0019] Set the SE-Dense module to include four layers. Each layer consists of a 1×1 convolution, a 3×3 convolution, and an SE attention mechanism. The 1×1 convolution is used to compress the number of channels and reduce the amount of computation. The 3×3 convolution is used to extract disease features. The SE attention mechanism is used to establish the connection between channels to improve the reusability of channels. The connection between layers in a dense manner improves the model's ability to express the scale of disease features;

[0020] The multi-layer information feature fusion module is set to include four 1×1 convolutions and three selective multi-scale feature fusion mechanisms, and the selective multi-scale feature fusion mechanisms are set;

[0021] The multi-layer information feature fusion module takes the output features of Layer1, Layer2, Layer3, and Layer4 of SE-DenseNet as input. First, 1×1 convolutions are used to compress the feature channels of each layer and filter out useless feature information to obtain feature information with 96 channels. Then, the compressed features of Layer3 and Layer4 are input into the first selective multi-scale feature fusion mechanism to obtain a fused feature of size 7×7×96. Subsequently, the fused feature of 7×7×96 and the compressed feature of Layer2 are input into the second selective multi-scale feature fusion mechanism to obtain a fused feature of size 14×14×96. Finally, the fused feature of 14×14×96 and the compressed feature of Layer1 are input into the third selective multi-scale feature fusion mechanism to obtain the final fused feature of size 28×28×96;

[0022] Set the GRU location information fusion classifier: The GRU location information fusion classifier consists of a 1×1 convolutional layer, two GRU recurrent neural networks, and a fully connected layer classifier;

[0023] The GRU location information fusion classifier compresses the features obtained by the multi-layer information feature fusion module into spatial information with a single channel through the 1×1 convolutional layer; the spatial information features with a single channel are flattened through the Flatten operation to obtain a one-dimensional vector. For the target regions with the same disease, the one-dimensional vector values have the same characteristics, and the targets with the same disease characteristics or multiple regions with the same disease are represented by target 1, target 2,..., target n; a GRU recurrent neural network is used to compress the one-dimensional information and memorize the neuron characteristics of the same disease. The input hidden state h0 of this GRU recurrent neural network is initialized to 0, and the compressed hidden state h1 has the same number of channels as the last layer of features extracted by the SE-DenseNet network; h1 and the one-dimensional feature vector obtained by the SE-DenseNet network are used as the hidden state and input value of the second GRU recurrent neural network respectively to obtain a fused output vector, and the result is input into the fully connected layer classifier to obtain the final class output.

[0024] The training of the crop disease image classification model includes the following steps:

[0025] Training of SE-DenseNet:

[0026] The crop disease images Input the SE-DenseNet network to obtain the outputs of Layer1, Layer2, Layer3, and Layer4;

[0027] Calculate the loss value L using the following formula f and optimize the network parameters,

[0028]

[0029] where f represents the recognition network SE-DenseNet, and y i represents the true label of the i-th data, and x i ′ represents the i-th input data in x′, and L f represents the loss value calculated by the SE-DenseNet model, and B represents the number of crop disease images input in each batch;

[0030] Training of the multi-layer information feature fusion module: The multi-layer information feature fusion module performs information processing and fusion on the multi-scale feature information of each layer of SE-DenseNet; The 7×7×248-dimensional feature information extracted from Layer4 and the 7×7×120-dimensional feature information extracted from Layer3 are subjected to feature selection and matching through a 1×1 convolutional kernel to make their feature channel numbers the same, and the compressed feature information is input into the selective multi-scale feature fusion mechanism for information fusion; The fused feature information is successively fused with the feature information output by Layer2 and Layer1 to obtain the finally output 28×28×96-dimensional feature information;

[0031] The four-layer output features of the SE-DenseNet network are passed through the multi-layer information feature fusion module to obtain fused features, and then the fused features and the results of Layer4 are input into the GRU location information fusion classifier to obtain class outputs. Finally, the weight parameters of the multi-layer information feature fusion module and the GRU location information fusion classifier are fine-tuned through the cross-entropy loss function. The calculation formula for the fine-tuning process is as follows:

[0032]

[0033] where π represents the overall model, and y i represents the true label of the i-th data, and x i represents the i-th input data in x, and L π represents the loss value calculated by the overall model, and B represents the number of crop disease images input in each batch;

[0034] Training of the GRU location information fusion classifier:

[0035] The fused features are compressed into 28×28×1 dimensional information through a 1×1 convolutional kernel to highlight the location information of multi-target diseases in each image. The fused features are flattened to obtain a 1×784 one-dimensional vector, which is input together with the 1×248 dimensional vector obtained by global average pooling (GAP) of the features of the last layer of SE-DenseNet into two GRU recurrent neural networks and a fully connected layer classifier to obtain the output of crop disease categories. The first GRU is used to memorize and compress the location information extracted by the fusion module, and the second GRU is used to fuse the location information and the feature information extracted by SE-DenseNet. Finally, the vector obtained by the second GRU is input into the fully connected layer classifier to obtain the category output.

[0036] The training of the SE-DenseNet includes the following steps:

[0037] For crop disease images of size 224×224×3, a feature map of size 56×56×24 is obtained through max pooling operations with convolutional kernel sizes of 7×7 and 3×3.

[0038] Input the first SE-Dense module and transition layer Layer1 to obtain features of size 28×28×96.

[0039] Input the second SE-Dense module and transition layer Layer2 to obtain features of size 14×14×112.

[0040] Input the third SE-Dense module and transition layer Layer3 to obtain features of size 7×7×120.

[0041] Input the fourth SE-Dense module Layer4 to obtain features of size 7×7×248, and through the global average pooling layer, obtain a 1×248 dimensional feature vector.

[0042] The setting of the selective multi-scale feature fusion mechanism includes the following steps:

[0043] Use the deep features as the weights of the attention mechanism to filter out the location information features related to crop disease recognition in the shallow features.

[0044] Adopt the EMA attention mechanism with cross-space learning ability for the deep features and the shallow features Since the scale size of the deep features is smaller than that of the shallow features, first perform bilinear interpolation on f high to transform it into an output feature map with the same dimension as f low The calculation formula is as follows: The calculation formula is as follows:

[0045] f high_out (x,y) = (1 - dx)(1 - dy)f high (x1,y1) + dx(1 - dy)f high (x2,y1) + (1 - dx)dyf high (x1,y2) + dxdyf high (x2,y2),

[0046] where (x1,y1), (x2,y2), (x1,y2) and (x2,y1) represent the coordinates of the four pixels closest to the new pixel coordinates, dx and dy represent the differences between the new pixel coordinates and the closest pixel coordinates, and f high is the pixel of the original image, and f high_out is the pixel of the interpolated image;

[0047] Take as the input of the EMA attention mechanism, use the groups operation to reshape some channels to the batch dimension, and group the channel dimension into C / / r sub - features, denoted as

[0048] Subsequently, use average pooling kernels of size (H,1) and (1,W) to calculate each grouped sub - feature along the horizontal and vertical coordinate directions. The output formula for the c - th channel with height H is:

[0049]

[0050] where x c is the c - th input feature map of f h ′ igh_out , W is the width of the feature map, is the output result of the c - th feature map in the height direction of the feature map; Similarly, the output formula for the c - th channel with width W is:

[0051]

[0052] where x c is the c - th input feature map of f h ′ igh_out , H is the height of the feature map, is the output result of the c - th feature map in the width direction of the feature map;

[0053] Connect the two - direction features z H and z W through Concat, and use a 1×1 - sized convolutional kernel with shared weights for transformation. The formula is as follows:

[0054] ρ = σ(P1([z H,z W ))

[0055] Among them, P1 represents a 1×1 point convolution represents the intermediate feature map where the spatial information is encoded in the horizontal and vertical directions, r represents the compression ratio factor, and σ represents the ReLU non-linear activation function;

[0056] Slice ρ along the spatial dimension into two separate tensors and Then use the sigmoid function to obtain the weight values in the horizontal and vertical directions. The formula is as follows:

[0057]

[0058] Among them, δ represents the sigmoid activation function, and the outputs and are the attention weights in the H direction and the W direction;

[0059] Multiply the feature map at the corresponding position by the attention weights in the two directions to obtain the 1×1 branch attention feature map. The calculation formula is as follows:

[0060]

[0061] where (i, j) is the coordinate of the i-th row and the j-th column, x t represents the input feature map of the t-th channel, represents the weight corresponding to the t-th channel in the H direction, represents the weight corresponding to the t-th channel in the W direction, y t represents the output feature map of the t-th channel. Therefore, the weighted sub-feature maps of C / / r channels are

[0062] The input f h ′ igh_out will be obtained through a 3×3 convolution operation to expand the receptive field of the feature space information,

[0063] G = σ(P3(f′ high_out ))

[0064] where P3 represents a 3×3 convolution represents the feature map with a higher receptive field, denoted as the 3×3 branch. Here, r represents the compression ratio factor, and σ represents the ReLU non-linear activation function;

[0065] Perform feature aggregation on the 1×1 branch attention feature map y and the 3×3 branch feature map G in the Gross-spatial learning manner. The calculation formula is as follows:

[0066] T = G·θ(GAP(GN(y))) + GN(y)·θ(GAP(G)),

[0067] where θ represents the softmax activation function, GN represents GroupNorm, GAP represents global average pooling, represents the output information after feature aggregation, y is the 1×1 branch attention feature map, and G is the 3×3 branch feature map;

[0068] After passing the aggregated feature T through the sigmoid function, the spatial attention weight value is obtained and fused with f low and the feature map f high_out The fusion formula is as follows:

[0069] f out = f low × δ(T) + f high_out ,

[0070] where δ represents the sigmoid activation function, represents the fused feature map, f low represents the shallow feature, and f high_out represents the value of the deep feature after bilinear interpolation.

[0071] Beneficial effects

[0072] A method for classifying crop disease images based on multi-layer information fusion according to the present invention proposes a multi-layer information fusion method based on a lightweight dense scale network compared with the prior art to achieve effective identification of crop diseases.

[0073] The method of the present invention extracts crop disease features at multiple scales through a dense network (SE-DenseNet) with a Squeeze-and-Excitation attention mechanism, and then fuses the feature information of each layer of the network through a multi-layer pyramid information feature fusion (MPIFF) module to reuse low-level to high-level semantic features and filter out background interference to obtain the location of the effective disease area. Finally, a GRU location fusion classifier (GRU-LIFC) is used to fuse the abstract features extracted by the SE-DenseNet and the target area location information to obtain the classification result. This fusion method reduces background interference and well focuses on the target tiny disease area, thereby improving the recognition effectiveness of the model.

[0074] On the 12 - class crop disease dataset constructed by the present invention, the recognition accuracy reaches 98.3%, the number of parameters is only 2.8M, and the computational amount is only 1.3G. Compared with the optimal Swin Transformer, the accuracy, precision, and recall are improved by 1.8%, 0.6%, and 6.1% respectively. This method has the characteristics of higher accuracy and fewer parameters, and can be used for effective and high - precision recognition of crop diseases in complex environments. Brief Description of the Drawings

[0075] Figure 1 It is the sequence diagram of the method of the present invention;

[0076] Figure 2 It is the schematic structural diagram of the crop disease image classification model involved in the present invention;

[0077] Figure 3 It is the schematic structural diagram of the SE - Dense module involved in the present invention;

[0078] Figure 4 It is the schematic structural diagram of the multi - layer information feature fusion module involved in the present invention;

[0079] Figure 5 It is the schematic structural diagram of the GRU position information fusion classifier involved in the present invention;

[0080] Figure 6a It is the comparison diagram of the loss value curves of two classifiers for 200 iterations;

[0081] Figure 6b It is the comparison diagram of the accuracy value curves of two classifiers for 200 iterations;

[0082] Figure 7 It is the analysis diagram of the confusion matrix of 12 - class crop disease data;

[0083] Figure 8 It is the visualization result diagram of Grad - CAM for crop disease data recognition. Detailed Embodiment

[0084] To have a further understanding and recognition of the structural features and achieved effects of the present invention, the following is a detailed description with preferred embodiments and attached drawings:

[0085] As Figure 1 shown, a method for classifying crop disease images based on multi - layer information fusion according to the present invention includes the following steps:

[0086] The first step, acquisition and pre - processing of crop disease images: Acquire crop disease data images and perform pre - processing.

[0087] Step 2: Construct a crop disease image classification model: As Figure 2 shown, construct a crop disease image classification model based on SE-DenseNet, a multi-layer information feature fusion module, and a GRU location information fusion classifier.

[0088] SE-DenseNet consists of densely connected SE-Dense modules, which enables the model to have diverse receptive fields, thus improving the expression ability for disease scale differences. The multi-layer information feature fusion module can effectively fuse feature maps with different scale sizes in each layer of SE-DenseNet to achieve feature reuse. In addition, taking the finally fused features as the target disease location information can improve the recognition effectiveness of the entire model. The GRU location information fusion classifier promotes the effectiveness and accuracy of the model in disease recognition by combining the features of the last layer of SE-DenseNet with the location information of the target disease area.

[0089] (1) Set that the crop disease image classification model includes SE-DenseNet, a multi-layer information feature fusion module, and a GRU location information fusion classifier;

[0090] Set that the input of SE-DenseNet is an image of 224×224×3. Through SE-DenseNet, multi-scale feature information extraction and fusion are performed. Layer1 outputs feature information with a dimension of 28×28×96, Layer2 outputs feature information with a dimension of 14×14×112, Layer3 outputs feature information with a dimension of 7×7×120, and Layer4 outputs feature information with a dimension of 7×7×248. The input of the multi-layer information feature fusion module is the features extracted by Layer1, Layer2, Layer3, and Layer4 of SE-DenseNet, and the output is feature information with a dimension of 28×28×96. The input of the GRU location information fusion classifier is the feature information with a dimension of 28×28×96 obtained by the multi-layer information feature fusion module and the 1×248-dimensional feature vector obtained by global average pooling of the output of the last layer of SE-DenseNet, which is 7×7×248, and the output is 12 types of crop disease types.

[0091] (2) Set SE-DenseNet: Set that SE-DenseNet includes four SE-Dense modules. Transition layers are connected between the four SE-Dense modules. The transition layer consists of a 3×3 convolution with a stride of 2, which is used to compress the number of channels and reduce the size of the feature map;

[0092] The set SE-Dense module consists of four layers, and each layer is composed of a 1×1 convolution, a 3×3 convolution, and an SE attention mechanism. The 1×1 convolution is used to compress the number of channels and reduce the computational amount. The 3×3 convolution is used to extract disease features. The SE attention mechanism is used to establish the connection between channels to improve the reusability of channels. The connection between layers in a dense manner improves the model's ability to express the scale of disease features.

[0093] Due to the large differences in the intra-class and inter-class scales of crop diseases collected and obtained in different environments and periods, the present invention designs a lightweight SE-DenseNet mainly composed of dense connection blocks SE-Dense modules for feature extraction. Figure 3 As shown, the SE-Dense module structure includes 4 layers. Each layer is composed of a 1×1 convolution block, a 3×3 convolution block, and an SE attention mechanism, and connects the features extracted from all previous layers in a Concat manner. This dense connection method alleviates the problem of gradient disappearance, enhances feature propagation, promotes feature reuse, and greatly reduces the number of parameters. In addition, different lines with multiple dense connections have different receptive fields, enhancing the diversity of the model's ability to extract features with different scales. We inserted an SE attention mechanism after each layer of convolution to improve the model's attention to disease features. The SE attention mechanism is a technique that focuses on the importance of each channel in the image feature map.

[0094] In the SE-DenseNet convolutional neural network, although more feature channel information is aggregated through dense connection, each channel of the feature map represents a specific feature information. Therefore, the contribution of different channels to the final task often varies. The SE attention mechanism dynamically adjusts the weight of each channel, enabling the network to focus on the features more critical for crop disease recognition, thereby improving the overall performance. By reusing Layer1 to Layer4, our method does not require the subsequent layer to aggregate more channel features than the previous layer as in existing dense networks, which greatly reduces the number of parameters brought by convolutional calculations.

[0095] (3) The crop disease features extracted by the convolutional neural network transition from low-level to high-level. The features extracted in its shallow layer mainly contain low-level details such as edges, textures, and shapes. As the number of network layers deepens, the receptive field becomes larger and the feature map output contains more and more abstract semantic information. In addition, in the recognition and classification network, the layer spatial size gradually decreases and the number of feature channels gradually increases. In order to better utilize the discriminant information contained in different network layers, the present invention designs a multi-layer information feature fusion module (Multi-layer pyramid information feature fusionmodule), such as Figure 4As shown. For the features obtained from different network layers, first, 1×1 convolutional kernels are used to screen the feature maps of different scales and match the feature channel dimensions to be consistent. Then, the semantic feature information of different scales is fused in sequence from high-level to low-level in a bottom-up order. The present invention adopts a Selective Multi-Scale Feature Fusion (SMFF) mechanism to combine the screened low-level features and high-level features. After the high-level features are expanded, scale adjustment is performed through bilinear interpolation, and then they are fused with the low-level features, thereby enhancing the model's ability to express crop disease features.

[0096] It is set that the multi-layer information feature fusion module includes four 1×1 convolutions and three selective multi-scale feature fusion mechanisms, and the selective multi-scale feature fusion mechanism is set.

[0097] Setting the selective multi-scale feature fusion mechanism includes the following steps:

[0098] A1) Using the deep features as the weights of the attention mechanism, screening out the position information features related to crop disease recognition in the shallow features;

[0099] Adopting the EMA attention mechanism with cross-space learning ability for the deep features and the shallow features Since the scale of the deep features is smaller than that of the shallow features, first, f is transformed into an output feature map with the same dimension as f through bilinear interpolation high low The calculation formula is as follows:

[0100] f high_out (x,y) = (1 - dx)(1 - dy)f high (x1,y1) + dx(1 - dy)f high (x2,y1) + (1 - dx)dyf high (x1,y2) + dxdyf high (x2,y2),

[0101] where, (x1,y1), (x2,y2), (x1,y2) and (x2,y1) represent the coordinates of the four pixels closest to the new pixel coordinates, dx and dy represent the differences between the new pixel coordinates and the closest pixel coordinates, f high is the original image pixel, and f high_out is the interpolated image pixel;

[0102] A2) Taking as the input of the EMA attention mechanism, using the groups operation to reshape some channels to the batch dimension, and grouping the channel dimension into C / / r sub-features, denoted as

[0103] Then, the average pooling kernels of size (H, 1) and (1, W) are used to calculate each sub-feature after grouping along the horizontal coordinate direction and the vertical coordinate direction. The output formula of the cth channel with a height of H is:

[0104]

[0105] Among them, x c f h ' igh_out The cth input feature map, W is the width of the feature map, is the output result of the cth feature map in the height direction of the feature map; similarly, the output formula of the cth channel with a width of W is:

[0106]

[0107] Among them, x c f h ' igh_out The cth input feature map, H is the height of the feature map, is the output result of the c-th feature map in the width direction of the feature map;

[0108] A3) Use Concat to convert z H and z W The two directional features are concatenated and transformed using a 1×1 convolution kernel with shared weights, as follows:

[0109] ρ=σ(P1([z H ,z W ])),

[0110] Among them, P1 represents 1×1 point convolution, Represents the intermediate feature map encoding spatial information in the horizontal and vertical directions, r represents the compression scale factor, and σ represents the ReLU nonlinear activation function;

[0111] A4) Split ρ into two separate tensors along the spatial dimension and Then use the sigmoid function to get the weight values in the horizontal and vertical directions. The formula is as follows:

[0112]

[0113] Where δ represents the sigmoid activation function, the output and is the attention weight in the H direction and the W direction;

[0114] A5) Multiply the feature map at the corresponding position by the attention weights in two directions to obtain the 1×1 branch attention feature map. The calculation formula is as follows:

[0115]

[0116] where (i,j) is the coordinate of the i-th row and the j-th column, x t represents the input feature map of the t-th channel, represents the weight corresponding to the t-th channel in the H direction, represents the weight corresponding to the t-th channel in the W direction, y t represents the output feature map of the t-th channel. Therefore, the sub-feature map after weighting for C / / r channels is

[0117] A6) The input f h ′ igh_out will be obtained through a 3×3 convolution operation to expand the receptive field of the feature space information,

[0118] G = σ(P3(f′ high_out )),

[0119] where P3 represents the 3×3 convolution, represents the feature map with a higher receptive field, denoted as the 3×3 branch. Here, r represents the compression ratio factor, and σ represents the ReLU non-linear activation function;

[0120] A7) Perform feature aggregation on the 1×1 branch attention feature map y and the 3×3 branch feature map G in the Gross-spatial learning manner. The calculation formula is as follows:

[0121] T = G·θ(GAP(GN(y))) + GN(y)·θ(GAP(G)),

[0122] where θ represents the softmax activation function, GN represents GroupNorm, and GAP represents global average pooling, represents the output information after feature aggregation, y is the 1×1 branch attention feature map, and G is the 3×3 branch feature map;

[0123] A8) After passing the aggregated feature T through the sigmoid function, obtain the spatial attention weight value and fuse it with f low and the feature map f high_out The calculation formula is as follows:

[0124] f out = f low ×δ(T) + f high_out ,

[0125] where δ represents the sigmoid activation function, represents the fused feature map, f low represents the shallow feature, f high_out represents the value of the deep feature after bilinear interpolation.

[0126] The multi-layer information feature fusion module takes the output features of Layer1, Layer2, Layer3, and Layer4 of SE-DenseNet as inputs. First, it uses 1×1 convolution to compress the feature channels of each layer and filter out useless feature information to obtain feature information with 96 channels. Then, the compressed features of Layer3 and Layer4 are input into the first selective multi-scale feature fusion mechanism to obtain a fused feature of size 7×7×96. Subsequently, the 7×7×96 fused feature and the compressed feature of Layer2 are input into the second selective multi-scale feature fusion mechanism to obtain a fused feature of size 14×14×96. Finally, the 14×14×96 fused feature and the compressed feature of Layer1 are input into the third selective multi-scale feature fusion mechanism to obtain the final fused feature of size 28×28×96.

[0127] (4) Set the GRU location information fusion classifier: The GRU location information fusion classifier consists of a 1×1 convolutional layer, two GRU recurrent neural networks, and a fully connected layer classifier.

[0128] The deep recognition network often uses global average pooling in the last layer to obtain feature vectors for final classification, but this is likely to destroy the spatial location information of the discriminant region. Especially for a crop disease image with multiple target disease regions, capturing this location information can improve feature diversity. Therefore, the present invention designs a GRU location information fusion classifier (GRU location information fusion classifier) to better fuse the target disease location information, thereby improving the image classification and recognition performance, as Figure 5 shown.

[0129] The GRU location information fusion classifier compresses the features obtained by the multi-layer information feature fusion module into single-channel spatial information through a 1×1 convolutional layer; flattens the single-channel spatial information features through a Flatten operation to obtain a one-dimensional vector. For the target regions with the same disease, the one-dimensional vector values have the same characteristics. The targets with the same disease characteristics or multiple identical disease regions are represented by target 1, target 2, …, target n; a GRU recurrent neural network is used to compress the one-dimensional information and memorize the neuron characteristics of the same disease. The input hidden state h0 of this GRU recurrent neural network is initialized to 0, and the number of channels of the compressed hidden state h1 is equal to that of the last layer of features extracted by the SE-DenseNet network; h1 and the one-dimensional feature vector obtained by the SE-DenseNet network are used as the hidden state and input value of the second GRU recurrent neural network respectively to obtain a fused output vector, and the result is input into the fully connected layer classifier to obtain the final class output.

[0130] Step 3, training of the crop disease image classification model: Use the preprocessed crop disease data images to train the crop disease image classification model. This patent adopts a two-stage training strategy. In the first stage, SE-DenseNet is trained, and in the second stage, the multi-layer information fusion module and the GRU location information fusion classifier are trained. This two-stage training strategy can reduce the consumption of computing resources, energy consumption, and training costs of computer devices. This strategy not only improves the performance of the model but also optimizes the training process, making the entire system more efficient and energy-saving.

[0131] (1) Training of SE-DenseNet:

[0132] Input the crop disease image into the SE-DenseNet network to obtain the outputs of Layer1, Layer2, Layer3, and Layer4.

[0133] B1) For crop disease images with a size of 224×224×3, since the images contain more redundant information, downsampling is performed through max pooling with convolutional kernel sizes of 7×7 and 3×3 to obtain a feature map with a size of 56×56×24. This process helps to reduce the redundant information in the images;

[0134] B2) Input the first SE-Dense module to aggregate multi-scale features and compress useless channel information through the transition layer (Layer1) to obtain features with a size of 28×28×96;

[0135] B3) Input the second SE-Dense module to aggregate multi-scale features and compress useless channel information through the transition layer (Layer2) to obtain features with a size of 14×14×112;

[0136] B4) Input the third - layer SE - Dense module to aggregate multi - scale features and compress useless channel information through the transition layer (Layer3), obtaining features of size 7×7×120.

[0137] B5) Input the fourth - layer SE - Dense module to aggregate multi - scale features (Layer4), obtaining features of size 7×7×248, and through the global average pooling layer, obtaining a 1×248 - dimensional feature vector.

[0138] (2) Calculate the loss value L using the following formula f and optimize the network parameters.

[0139]

[0140] where f represents the recognition network SE - DenseNet, y i represents the true label of the i - th data, x i ′ represents the i - th input data in x′, L f represents the loss value calculated by the SE - DenseNet model, and B represents the number of crop disease images input in each batch.

[0141] (3) Training of the multi - layer information feature fusion module: The multi - layer information feature fusion module processes and fuses the multi - scale feature information of each layer of SE - DenseNet; The 7×7×248 - dimensional feature information extracted from Layer4 and the 7×7×120 - dimensional feature information extracted from Layer3 are subjected to feature selection and matching through a 1×1 - sized convolutional kernel to make their feature channel number dimensions the same, and the compressed feature information is input into the selective multi - scale feature fusion mechanism for information fusion; The fused feature information is successively fused with the feature information output from Layer2 and Layer1, thereby obtaining the finally output 28×28×96 - dimensional feature information;

[0142] The four - layer output features of the SE - DenseNet network are passed through the multi - layer information feature fusion module to obtain fused features, and then the fused features and the results of Layer4 are input into the GRU location information fusion classifier to obtain class outputs. Finally, the weight parameters of the multi - layer information feature fusion module and the GRU location information fusion classifier are fine - tuned through the cross - entropy loss function. The calculation formula for the fine - tuning process is as follows:

[0143]

[0144] where π represents the overall model, y i represents the true label of the i - th data, x i represents the i - th input data in x, Lπ Loss value calculated by the overall model, and B represents the number of crop disease images input in each batch.

[0145] (4) Training of the GRU location information fusion classifier:

[0146] The fused features are compressed into 28×28×1 dimensional information through a 1×1 convolutional kernel to highlight the location information of multi-target diseases in each image; the fused features are flattened to obtain a 1×784 one-dimensional vector, which is input together with the 1×248 dimensional vector obtained by global average pooling GAP of the features of the last layer of SE-DenseNet into two GRU recurrent neural networks and a fully connected layer classifier to obtain the output of the crop disease category; the first GRU is used to memorize and compress the location information extracted by the fusion module, and the second GRU is used to fuse the location information and the feature information extracted by SE-DenseNet; finally, the vector obtained by the second GRU is input into the fully connected layer classifier to obtain the category output.

[0147] Fourth step, obtaining the crop disease images to be classified: Obtain the crop disease images to be classified and perform preprocessing.

[0148] Fifth step, obtaining the classification results of crop disease images: Input the preprocessed crop disease images to be classified into the trained crop disease image classification model to obtain the classification results of crop disease images.

[0149] To show that the fusion classifier constructed using GRU is superior to the most commonly used fully connected layer classifier method currently, an experimental comparison of the training process was carried out on the 12-class crop disease data in natural scenes constructed. As Figure 6a and Figure 6b shown, although the classifier constructed using the fully connected layer and the classifier proposed in this patent achieved similar training losses and changes in training accuracy, the method of this invention has more stable test losses and test accuracy on the test set, and the test accuracy is higher. This benefits from the memory function of the GRU recurrent neural network, which has better ability to capture long-term dependence relationships.

[0150] For the training strategy adopted in this invention, the confusion matrix results of the recognition of 12-class crop disease data were plotted on the test data set, as Figure 7 shown. The method of this invention achieved good recognition performance on the crop disease data obtained in 12 natural environments. Especially for the recognition accuracy of Rfs, Rbs and Rsb diseases reached 100%. These types of diseases show dense distribution on the leaves, and the method of this invention can well capture the global information of the disease location, prompting the model to learn more features, thus improving the recognition performance.

[0151] To demonstrate that the designed multi-layer information fusion module and GRU location information fusion classifier can improve the model's ability to focus on the location of crop disease target areas in natural scenes, the present invention uses Grad-CAM heatmaps to visualize three common types of crop diseases, such as Figure 8 shown. As can be seen from Figure 8 , the method described in the present invention can well focus on the crop disease areas. In addition, for smaller crop disease areas, the method described in the present invention can also accurately focus on them, which is very important for the effectiveness of identifying crop diseases in natural environments.

[0152] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the principles described in the above embodiments and the specification are only the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection required by the present invention is defined by the appended claims and their equivalents.

Claims

1. A crop disease image classification method based on multi-layer information fusion, characterized in that: The following steps are involved: 11) Acquisition and preprocessing of crop disease images: Acquisition of crop disease data images and preprocessing; 12) Constructing a crop disease image classification model: Constructing a crop disease image classification model based on SE-DenseNet, a multi-layer information feature fusion module, and a GRU position information fusion classifier; The crop disease image classification model is set to include SE-DenseNet, multi-layer information feature fusion module and GRU position information fusion classifier; Setting up SE-DenseNet: SE-DenseNet is assumed to include four layers of SE-Dense modules. Each of the four layers of SE-Dense modules is connected to a transition layer. The transition layer consists of a 3×3 convolution with a stride of 2, which is used to compress the number of channels and reduce the size of the feature map. The SE-Dense module is set to consist of four layers, each of which consists of 1×1 convolution, 3×3 convolution, and SE attention mechanism. The 1×1 convolution is used to compress the number of channels and reduce the amount of computation. The 3×3 convolution is used to extract disease features. The SE attention mechanism is used to establish connections between channels to improve channel reusability. Dense connections between layers improve the model's ability to express disease feature scales. The multi-layer information feature fusion module is set to include four 1×1 convolutions and three selective multi-scale feature fusion mechanisms, and the selective multi-scale feature fusion mechanism is set; Setting up the GRU position information fusion classifier: The GRU position information fusion classifier consists of a 1×1 convolutional layer, two GRU recurrent neural networks, and a fully connected layer classifier; 13) Training of crop disease image classification model: Using pre-processed crop disease data images, the crop disease image classification model is trained; 14) Acquisition of crop disease images to be classified: Acquisition of crop disease images to be classified and preprocessing; 15) Obtaining the crop disease image classification results: The pre-processed crop disease images to be classified are input into the trained crop disease image classification model to obtain the crop disease image classification results.

2. The crop disease image classification method based on multi-layer information fusion according to claim 1, characterized in that: The construction of the crop disease image classification model comprises the following steps: 21) Assume that the input of SE-DenseNet is a 224×224×3 image, and extract multi-scale feature information through SE-DenseNet for processing and fusion. Layer1 outputs 28×28×96 dimensional feature information, Layer2 outputs 14×14×112 dimensional feature information, Layer3 outputs 7×7×120 dimensional feature information, and Layer4 outputs 7×7×248 dimensional feature information; the input of the multi-layer information feature fusion module is the features extracted by Layer1, Layer2, Layer3 and Layer4 of SE-DenseNet, and the output is 28×28×96 dimensional features; the input of the GRU position information fusion classifier is the 28×28×96 dimensional features obtained by the multi-layer information feature fusion module and the 7×7×248 output of the last layer of SE-DenseNet. After global average pooling, a 1×248 dimensional feature vector is obtained, and the output is 12 types of crop diseases; 22) The multi-layer information feature fusion module takes the output features of Layer 1, Layer 2, Layer 3 and Layer 4 of SE-DenseNet as input. It first uses 1×1 convolution to compress the feature channels of each layer and filters out useless feature information to obtain feature information with a channel number of 96. Then, the compressed features of Layer 3 and Layer 4 are input into the first selective multi-scale feature fusion mechanism to obtain a fused feature of size 7×7×96. Subsequently, the fused feature of 7×7×96 and the compressed feature of Layer 2 are input into the second selective multi-scale feature fusion mechanism to obtain a fused feature of size 14×14×96. Finally, the fused feature of 14×14×96 and the compressed feature of Layer 1 are input into the third selective multi-scale feature fusion mechanism to obtain the final fused feature of size 28×28×96. 23) The GRU position information fusion classifier compresses the features obtained by the multi-layer information feature fusion module into the spatial information of a single channel through a 1×1 convolutional layer; the spatial information features of the single channel are flattened through the Flatten operation to obtain a one-dimensional vector. For the target areas with the same disease, the one-dimensional vector values have the same characteristics. Target 1, target 2, ..., target n are used to represent targets with the same disease characteristics or multiple targets with the same disease areas; a GRU recurrent neural network is used to compress the one-dimensional information and memorize the neuronal characteristics of the same disease. The input hidden state h0 of the GRU recurrent neural network is initialized to 0, and the compressed hidden state h1 is equal to the number of feature channels of the last layer extracted by the SE-DenseNet network; h1 and the one-dimensional feature vector obtained by the SE-DenseNet network are used as the hidden state and input value of the second GRU recurrent neural network respectively to obtain the fused output vector, and the result is input to the fully connected layer classifier to obtain the final category output.

3. The crop disease image classification method based on multi-layer information fusion according to claim 1, characterized in that: The training of the crop disease image classification model includes the following steps: 31) SE-DenseNet training: Crop disease images Input the SE-DenseNet network and get the output of Layer1, Layer2, Layer3, and Layer4; 32) Use the following formula to calculate the loss value L f and optimize network parameters, Among them, f represents the recognition network SE-DenseNet, y i represents the true label of the i-th data, x i ′ represents the i-th input data in x′, L f represents the loss value calculated by the SE-DenseNet model, and B represents the number of crop disease images input in each batch; 33) Training of the multi-layer information feature fusion module: The multi-layer information feature fusion module processes and fuses the multi-scale feature information of each layer of SE-DenseNet. The 7×7×248 dimensional feature information extracted by Layer4 and the 7×7×120 dimensional feature information extracted by Layer3 are selected and matched using a 1×1 convolution kernel to make their feature channels have the same number of dimensions. The compressed feature information is then input into the selective multi-scale feature fusion mechanism for information fusion. The fused feature information is then fused with the feature information output by Layer2 and Layer1 in turn, resulting in a final output of 28×28×96 dimensional feature information. The four-layer output features of the SE-DenseNet network are fused through the multi-layer information feature fusion module to obtain fused features. Then, the fused features and the results of Layer 4 are input into the GRU position information fusion classifier to obtain the category output. Finally, the weight parameters of the multi-layer information feature fusion module and the GRU position information fusion classifier are fine-tuned through the cross entropy loss function. The calculation formula used in the fine-tuning process is as follows: Among them, π represents the overall model, y i represents the true label of the i-th data, x i represents the i-th input data in x, L π represents the loss value calculated by the overall model, and B represents the number of crop disease images input in each batch; 34) Training of GRU position information fusion classifier: The fused features are compressed into 28×28×1 dimensional information using a 1×1 convolution kernel to highlight the location information of multiple target diseases in each image. The fused features are flattened to obtain a 1×784 one-dimensional vector, which is input into two GRU recurrent neural networks and a fully connected layer classifier together with the 1×248 dimensional vector obtained by global average pooling (GAP) of the last layer features of SE-DenseNet to obtain the output of crop disease categories. The first GRU is used to memorize and compress the location information extracted by the fusion module, and the second GRU is used to fuse the location information with the feature information extracted by SE-DenseNet. Finally, the vector obtained by the second GRU is input into the fully connected layer classifier to obtain the category output.

4. The crop disease image classification method based on multi-layer information fusion according to claim 3 is characterized in that: The training of SE-DenseNet includes the following steps: 41) For a crop disease image of size 224×224×3, a feature map of size 56×56×24 is obtained by performing a maximum pooling operation with a convolution kernel size of 7×7 and 3×3; 42) Input the first layer SE-Dense module and transition layer Layer1 to obtain a feature size of 28×28×96; 43) Input the second layer SE-Dense module and transition layer Layer2 to obtain a feature size of 14×14×112; 44) Input the third layer SE-Dense module and transition layer Layer3 to obtain a feature size of 7×7×120; 45) Input the fourth layer SE-Dense module Layer4 to obtain a feature of size 7×7×248, and pass it through the global average pooling layer to obtain a 1×248 dimensional feature vector.

5. The crop disease image classification method based on multi-layer information fusion according to claim 2, characterized in that: The setting of the selective multi-scale feature fusion mechanism includes the following steps: 51) Using deep features as the weight of the attention mechanism, the location information features related to crop disease identification are screened out from the shallow features; Adopting EMA attention mechanism with cross-space learning ability, for deep features and shallow features Since the scale of deep features is smaller than that of shallow features, firstly, f high Convert to and f low Output feature maps with the same dimensions The calculation formula is as follows: f high_out (x,y)=(1-dx)(1-dy)f high (x1,y1)+dx(1-dy)f high (x2,y1)+(1-dx)dyf high (x1,y2)+dxdyf high (x2,y2), Among them, (x1, y1), (x2, y2), (x1, y2) and (x2, y1) represent the coordinates of the four pixels closest to the new pixel coordinates, dx and dy represent the difference between the new pixel coordinates and the nearest pixel coordinates, and f high is the original image pixel, f high_out Interpolated image pixels; 52) As the input of the EMA attention mechanism, the groups operation is used to reshape some channels to the batch dimension, and the channel dimension is grouped into C / / r sub-features, recorded as Then, the average pooling kernels of size (H, 1) and (1, W) are used to calculate each sub-feature after grouping along the horizontal coordinate direction and the vertical coordinate direction. The output formula of the cth channel with a height of H is: Among them, x c f′ high_out The cth input feature map, W is the width of the feature map, is the output result of the cth feature map in the height direction of the feature map; similarly, the output formula of the cth channel with a width of W is: Among them, x c f′ high_out The cth input feature map, H is the height of the feature map, is the output result of the c-th feature map in the width direction of the feature map; 53) Use Concat to convert z H and z W The two directional features are concatenated and transformed using a 1×1 convolution kernel with shared weights, as follows: ρ=σ(P1([z H ,z W ])), Among them, P1 represents 1×1 point convolution, Represents the intermediate feature map encoding spatial information in the horizontal and vertical directions, r represents the compression scale factor, and σ represents the ReLU nonlinear activation function; 54) Split ρ into two separate tensors along the spatial dimension and Then use the sigmoid function to get the weight values in the horizontal and vertical directions. The formula is as follows: Where δ represents the sigmoid activation function, the output and is the attention weight in the H direction and the W direction; 55) Multiply the feature map of the corresponding position and the attention weights of the two directions to obtain a 1×1 branch attention feature map. The calculation formula is as follows: Where (i, j) is the i-th row and j-th column coordinate, x t represents the t-th channel input feature map, Represents the weight corresponding to the t-th channel in the H direction, Represents the weight corresponding to the t-th channel in the W direction, y t Represents the output feature map of the tth channel, and the weighted sub-feature map of C / / r channels is 56) Input f′ high_out The 3×3 convolution operation will be used to obtain Used to expand the receptive field of feature space information, Where P3 represents 3×3 convolution, Represents a feature map with higher sensitivity, recorded as a 3×3 branch, where r represents the compression scale factor and σ represents the ReLU nonlinear activation function; 57) Perform Gross-spatial learning on the 1×1 branch attention feature map y and the 3×3 branch feature map G to perform feature aggregation. The calculation formula is as follows: T=G·θ(GAP(GN(y)))+GN(y)·θ(GAP(G)), Where θ represents the softmax activation function, GN represents GroupNorm, and GAP represents global average pooling. Represents the output information after feature aggregation, y is a 1×1 branch attention feature map, and G is a 3×3 branch feature map; 58) The aggregated feature T is passed through the sigmoid function to obtain the spatial attention weight value and combined with f low and feature map f high_out The calculation formula for fusion is as follows: f out =f low ×δ(T)+f high_out , Where δ represents the sigmoid activation function, Represents the fused feature map, f low represents shallow features, f high_out Represents the value of the deep feature after bilinear interpolation.

Images