Apple leaf disease classification method based on region-aware and lightweight hybrid transformer network

Abstract

The application discloses an apple leaf disease classification method based on region perception and a lightweight hybrid Transformer network. Firstly, a region extraction and fine-grained convolution coding module is used to suppress the interference of low-frequency background and dynamic light, and to adaptively perceive the leaf region and the disease region. Secondly, a lightweight Transformer block with a parallel cascading structure is designed to extract multi-scale texture and edge features, and a jump connection mode is used to realize adaptive selection of the optimal feature scale, mining of long-distance dependency relationship, and cross-scale information interaction and fusion. Then, an adaptive learning-based feature weighting, reverse residual mechanism and channel attention enhancement network are used to enhance the attention degree of the region feature information, so that rich texture detail features of the apple leaf region are extracted. Finally, a weighted cross-entropy loss function and a focal loss function are jointly used to improve the generalization ability of the network. The experimental results verify the effectiveness of the application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of agriculture and computer vision analysis, and in particular, especially relates to an apple leaf disease classification method based on regional perception and a lightweight mixed Transformer network. BACKGROUND

[0002] China is the world's largest apple producer and consumer, with planting area and yield accounting for more than 40% of the world's total output. Apple yield and quality have an important impact on the income of fruit growers and the export of the fruit industry. However, the complex and variable growth environment of apple trees makes the prevention and control of leaf diseases a difficult problem for apple growers. At the same time, traditional manual identification methods not only take time and effort, but also have obvious insufficient accuracy, which can easily lead to the large-scale spread of apple tree leaf diseases and inevitably cause economic losses to fruit growers. In this case, accurate identification and classification of apple leaf diseases are of great significance to ensure the quality and safety of agricultural products and meet the growing demand for green agriculture and food safety.

[0003] At present, China's agricultural modernization is facing the urgent need for digital transformation and upgrading. Intelligent prevention and control of diseases, as an important part of smart agriculture, can effectively improve the efficiency and accuracy of disease diagnosis and help fruit growers take effective prevention and control measures in a timely manner. Therefore, apple leaf disease automatic classification aims to quickly and accurately locate and classify various diseases (such as black spot disease, spot disease, etc.) on apple leaves through digital image processing, computer vision and artificial intelligence technology, so as to realize real-time monitoring and accurate prediction of large-scale apple orchards.

[0004] According to the different overall technical framework, the existing apple leaf disease automatic classification methods mainly include three categories, i.e., apple leaf disease classification method based on traditional image processing and machine learning, apple leaf disease classification method based on deep learning, and apple leaf disease classification method based on transfer learning.

[0005] Firstly, the apple leaf disease classification method based on traditional image processing and machine learning mainly relies on classic digital image processing techniques to extract features such as color, texture and shape of apple leaves, and then uses machine learning algorithms such as support vector machine and decision tree for classification. However, this method has strong dependence on feature extraction and performs poorly in complex backgrounds.

[0006] Secondly, the apple leaf disease classification method based on deep learning aims to use deep convolutional neural networks (such as ResNet, VGG, etc.), generative adversarial networks and other models to identify various apple leaf diseases, and has achieved better performance than traditional classification methods. Liu et al. proposed an apple leaf disease identification method based on AlexNet neural network, which can identify 4 common apple leaf diseases. Yan et al. replaced the fully connected layer with a global average pooling layer to reduce parameters and added a batch normalization layer to improve convergence speed, thus proposing an apple leaf disease classification method (AICNN) based on an improved VGG16 model, which can handle 3 common apple leaf diseases: scab, downy mildew and cedar rust. Ding et al. used a dual attention mechanism to suppress the interference of complex background on the recognition process, combined class balancing technology with focal loss function to reduce the adverse effects of unbalanced data set on classification accuracy, and then proposed an apple leaf disease classification method based on RFCA ResNet. Li et al. designed a vegetation index selection method using saliency attention mechanism, used a multi-label ReliefF feature selection algorithm to realize automatic selection of vegetation index, and then constructed an apple disease area multi-label classification model AMMFNet, which can combine channel attention mechanism to explore the complementarity between RGB and multispectral image data. Li Ling et al. modified the size of the fully connected layer of the output layer of the MobileNet network model, and proposed an apple tree leaf disease classification method based on MobileNet. Shen Hengyu used Keras to build a MobileNetV3 neural network, and then adjusted the network structure to make it suitable for apple tree leaf disease classification and identification tasks, and then introduced hybrid attention mechanism and channel attention mechanism, and then proposed an apple tree leaf disease classification method based on an improved MobileNetV3 model. Liang Qianqian et al. designed a learnable filter layer using Fourier transform, replaced the multi-head attention module of the original MobileViT model, and then used deep convolutional layers and residual structures to enhance the model's feature learning ability for disease images, thus proposing an apple leaf disease classification and identification method based on an improved lightweight network MobileViT. Although the above methods have achieved high classification accuracy on specific data sets, in actual applications, the image data set obtained in the orchard usually has complex background, unbalanced disease class distribution, dynamic scene changes (such as leaf shading, fruit shading, diverse lighting and weather conditions, etc.), which affects the generalization ability of the above methods, and it is still difficult to meet the needs of practical applications.

[0007] Finally, due to the need for a large amount of data for training of deep learning models, transfer learning has become an effective means. Therefore, the apple leaf disease classification method based on transfer learning aims to use pre-trained models (such as VGG16, ResNet50, Inception V3, etc.) and transfer learning technology to detect and classify apple tree leaf diseases. Chen Hao et al. proposed an apple tree leaf disease image classification method based on transfer learning and VGG19 network through parameter transfer and model fine-tuning, which achieved good accuracy. Zhang et al. introduced a coordinate attention mechanism and a weight adaptive multi-scale feature fusion module into the ResNet-50 model, enhancing the feature extraction capability of the original ResNet-50 model, and using transfer learning and online data enhancement to improve the generalization ability of the model, thereby proposing an apple tree leaf disease classification method based on an improved ResNet-50 neural network. Zhou Hongwei et al. migrated the model weights trained on ImageNet to a new model with a redesigned output layer, and then trained the new model using apple tree leaf disease sample data, and disclosed an apple tree leaf disease classification and recognition method based on transfer learning. However, the data of apple leaf diseases is often limited and lacks diversity, and the lack of sufficient labeled data may lead to overfitting or poor generalization of this method. At the same time, changes in crop growth conditions (such as light, humidity, etc.) may make some features unstable in different environments, affecting the robustness and adaptability of the transfer model.

[0008] In summary, to achieve accurate automatic classification of apple leaf diseases is still a very challenging task. At present, there is no apple leaf disease automatic classification method that can effectively resist complex background and dynamic light interference, has high accuracy, good robustness, strong generalization ability and low computational complexity. SUMMARY

[0009] According to the technical problems mentioned in the above background art, a region-aware and lightweight hybrid Transformer network apple leaf disease classification method is provided. The present application can effectively resist complex background and dynamic light interference, and is a region-aware and lightweight hybrid Transformer network apple leaf disease classification method with high accuracy, good robustness, strong generalization ability and low computational complexity.

[0010] The technical means adopted by the present application are as follows:

[0011] The apple leaf disease classification method of the region-aware and lightweight hybrid Transformer network comprises the following steps:

[0012] Step 1: Establish and initialize a deep convolutional neural network N used for apple leaf disease classification apple; the deep convolutional neural network N apple comprises a target region perception subnetwork N obj , an image fine-grained feature extraction subnetwork N stem , a feature fusion subnetwork N fusion , four image local feature extraction subnetworks, respectively denoted as N local1 , N local2 , N local3 , N local4 , three image global feature extraction subnetworks, respectively denoted as N global1 , N global2 , N global3 , three local feature and global feature fusion subnetworks, respectively denoted as N Merge1 , N Merge2 , and N Merge3 , a channel attention module M ECA , and a classification head module M classifier ;

[0013] Step 2: input the training set of apple leaf disease images and the manually labeled label set C label , train the deep convolutional neural network N apple , obtain the trained deep convolutional neural network N apple ; the training set has N testImg images, and let the iteration number iter

[0014] Step 3: input the apple leaf disease image I to be processed, the color space of the apple leaf disease image I to be processed is RGB; according to the trained deep convolutional neural network N apple obtained in the step 2, the disease category of the apple leaf disease image I to be processed is predicted.

[0015] Further, the step 1 comprises the following steps:

[0016] Step 11: establish a target region perception subnetwork N obj and initialize it; the target region perception subnetwork N obj contains: 1 target region extraction module M extraction and 1 target region encoding module M stride1 ; the M extraction includes: 1 adaptive contrast enhancement module M conenc , 1 image segmentation module M seg based on the maximum inter-class variance, 1 morphological closing operation module M morph based on an elliptical structural element with a size of (10, 10);

[0017] the Mstride1 comprises 3 groups of convolution operations, respectively wherein, comprises 1 group of 18 convolution kernels with a size of 1x1, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; comprises 1 group of 18 convolution kernels with a size of 3x3 and a group number of 18, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; comprises 1 group of 32 convolution kernels with a size of 1x1, and 1 layer of BatchNorm2d layer;

[0018] Step 12: establishing an image fine-grained feature extraction subnetwork N stem and initializing it; the image fine-grained feature extraction subnetwork N stem includes a convolution layer Conv stem , a BatchNorm2d data normalization layer, and a ReLU6 activation function layer; the convolution layer Conv stem includes 32 convolution kernels with a size of 3x3, and each convolution kernel performs convolution operation with a step of 1 pixel;

[0019] Step 13: establishing a feature fusion subnetwork N fusion and initializing it; the feature fusion subnetwork N fusion includes 2 learnable weight coefficients w1 and w2;

[0020] Step 14: establishing an image local feature extraction subnetwork N local1 and initializing it; the image local feature extraction subnetwork N local1 includes 3 groups of convolution operations; the convolution operations are respectively the comprises 1 group of 192 convolution kernels with a size of 1x1; comprises 1 group of 192 convolution kernels with a size of 3x3 and a group number of 192, and each convolution kernel performs convolution operation with a step of 2 pixels, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; comprises 1 group of 64 convolution kernels with a size of 1x1, and 1 layer of BatchNorm2d layer;

[0021] Step 15: establishing an image global feature extraction subnetwork N global1 and initializing it; the image global feature extraction subnetwork N global1 comprises 1 group of pooling operations and 1 self-attention module the pooling operations The image local feature extraction subnetwork N comprises three groups of convolution operations, respectively

[0022] Step 16: Establishing an image local feature extraction subnetwork N local2 and initializing the same; the image local feature extraction subnetwork N local2 comprises three groups of convolution operations, respectively The image local feature extraction subnetwork N comprises one group of 384 convolution kernels with a size of 1x1; comprises one group of 384 convolution kernels with a size of 3x3 and a group number of 384, and each convolution kernel performs convolution operation with a step length of 2 pixels, one BatchNorm2d data normalization layer, and one ReLU6 activation function layer; comprises one group of 128 convolution kernels with a size of 1x1 and one BatchNorm2d layer;

[0023] Step 17: Establishing an image global feature extraction subnetwork N global2 and initializing the same; the image global feature extraction subnetwork N global2 comprises one group of pooling operations and one self-attention module The image global feature extraction subnetwork N comprises one layer of average pooling operation with a size of 2x2 and a step length of 2, and one layer of maximum pooling operation with a size of 2x2 and a step length of 2;

[0024] Step 18: Establishing an image global feature extraction subnetwork N global3 and initializing the same; the image global feature extraction subnetwork N global3 comprises one group of pooling operations and one self-attention module The image global feature extraction subnetwork N comprises one layer of average pooling operation with a size of 2x2 and a step length of 2, and one layer of maximum pooling operation with a size of 2x2 and a step length of 2;

[0025] Step 19: Establishing a local feature and global feature fusion subnetwork N Merge1 and initializing the same; the local feature and global feature fusion subnetwork N Merge1 comprises one group of 128 convolution kernels with a size of 1x1, and each convolution kernel performs convolution operation with a step length of 1 pixel;

[0026] Step 110: Establishing an image local feature extraction subnetwork N local3 and initializing the same; the image local feature extraction subnetwork N local3 comprises three groups of convolution operations, respectively The contains 1 group of 960 convolution kernels with a size of 1x1; contains 1 group of 960 convolution kernels with a size of 3x3 and a group number of 960, and each convolution kernel performs convolution operation with a step of 2 pixels, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; contains 1 group of 128 convolution kernels with a size of 1x1, and 1 layer of BatchNorm2d layer;

[0027] Step 111: Establishing a local feature and global feature fusion subnetwork N Merge2 and initializing it; the local feature and global feature fusion subnetwork N Merge2 contains 1 group of 256 convolution kernels with a size of 1x1, and each convolution kernel performs convolution operation with a step of 1 pixel;

[0028] Step 112: Establishing an image local feature extraction subnetwork N local4 and initializing it; the image local feature extraction subnetwork N local4 contains 3 groups of convolution operations, respectively The contains 1 group of 1920 convolution kernels with a size of 1x1; contains 1 group of 1920 convolution kernels with a size of 3x3 and a group number of 1920, and each convolution kernel performs convolution operation with a step of 2 pixels, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; contains 1 group of 256 convolution kernels with a size of 1x1 and 1 BatchNorm2d layer;

[0029] Step 113: Establishing a local feature and global feature fusion subnetwork N Merge3 and initializing it; the local feature and global feature fusion subnetwork N Merge3 contains 1 group of 960 convolution kernels with a size of 1x1, and each convolution kernel performs convolution operation with a step of 1 pixel;

[0030] Step 114: Establishing a channel attention module M ECA and initializing it; the channel attention module M ECA contains 1 global average pooling layer, 1 adaptive one-dimensional convolution layer, 1 Sigmoid activation function layer, and 1 matrix Hadamard product layer;

[0031] Step 115: Establishing a classification head module M classifier and initializing it; the classification head module Mclassifier The global average pooling layer comprises one global average pooling layer and one classifier.

[0032] The global average pooling layer pools the feature map by a factor; the classifier comprises one Hardwish activation function layer, one Dropout layer with a probability of 0.3, and one fully connected layer with an input unit of 960 and an output unit of 4.

[0033] Further, the step 2 comprises the following steps:

[0034] Step 21: Take the ind-th image T from the training set ind and its corresponding labeled label

[0035] Step 22: Perform horizontal flip operation with a probability of 0.4 and 20° random rotation operation on the image T ind to realize data enhancement of the image T ind ;

[0036] Step 23: Process the image T obj according to the target region perception subnetwork N ind to obtain the feature map F obj ;

[0037] Step 24: Process the image T according to the image fine-grained feature extraction subnetwork N stem to obtain the feature map F stem ;

[0038] Step 25: Calculate the feature map F fusion according to the feature fusion subnetwork N fusion ;

[0039] Step 26: Process F local1 according to the image local feature extraction subnetwork N fusion to obtain the feature map F local1 ;

[0040] Step 27: Process F global1 according to the image global feature extraction subnetwork N fusion to obtain the feature map F global1 ;

[0041] Step 28: Process F local2 according to the image local feature extraction subnetwork N local1 to obtain the feature map F local2 ;

[0042] Step 29: Process F global2 according to the image global feature extraction subnetwork N local1 to obtain the feature map F global2 ;

[0043] Step 210: processing F global3 according to the image global feature extraction subnetwork N local2 , to obtain feature map F global3 ;

[0044] Step 211: connecting F local2 and F global1 along the channel direction, and then processing through the local feature and global feature fusion subnetwork N Merge1 , to obtain feature map F merge1 ;

[0045] Step 212: processing F local3 according to the image local feature extraction subnetwork N merge1 , to obtain feature map F local3 ;

[0046] Step 213: connecting F local3 and F global2 along the channel direction, and then processing through the local feature and global feature fusion subnetwork N Merge2 , to obtain feature map F merge2 ;

[0047] Step 214: processing F local4 according to the image local feature extraction subnetwork N merge2 , to obtain feature map F local4 ;

[0048] Step 215: connecting F local4 and F global3 along the channel direction, and then processing through the local feature and global feature fusion subnetwork N Merge3 , to obtain feature map F merge3 ;

[0049] Step 216: according to the channel attention module M ECA , calculating feature map F merge3 according to formula (1), to obtain feature map F ECA ;

[0050]

[0051] wherein, GAP(·) represents the global average pooling operation, Conv 1×1 (·) represents one-dimensional convolution operation, δ(·) represents the Sigmoid function, and represents the matrix Hadamard product.

[0052] Step 217: using the classification head module M classifier to process FECA The processing is performed to calculate the image T ind The predicted probability of each category;

[0053] Step 218: In the predicted probability of each category, the category with the highest probability is selected as the predicted category of the image T ind , and the probability thereof is taken as the predicted probability of the image T ind ;

[0054] Step 219: Let ind <- ind + 1, if ind > N testImg , go to step 220, otherwise return to step 21;

[0055] Step 220: Calculate the predicted loss value L total of the training set:

[0056]

[0057] L total = ζ1×L crossEntropy + ζ2×L focal ;

[0058] The L focal represents the average focal loss value, L crossEntropy represents the average cross-entropy loss value, α j and ξ are hyperparameters, k represents the number of disease categories contained in the training set, p ij represents the probability that the model predicts that the i-th image of the training set belongs to the j-th disease category, y i represents the true label of the i-th image of the training set, represents the inverse frequency weighted weight coefficient of the y i th disease, represents the probability that the model predicts that the i-th image of the training set belongs to the y i th disease, N j represents the number of images belonging to the j-th disease in the training set, N yi represents the number of images belonging to the y i th disease in the training set, ζ1 and ζ2 represent preset weight coefficients;

[0059] Step 221: Let iter <- iter + 1, if the number of iterations iter > Total_iter, the trained deep convolutional neural network N apple is obtained, go to step 3, otherwise use the back-propagation algorithm based on the stochastic gradient descent method and the predicted loss L total to update N applethe parameter Total _ iter, let the counter ind <- 1, and go to step 21 to reprocess all images in the training set, wherein Total _ iter represents a preset iteration number.

[0060] Further, the step 23 further comprises the following steps:

[0061] Step 231: extracting a target region according to a target region extraction module M extraction Convolution operation is performed on the image T ind to obtain T';

[0062] Step 232: encoding the target region according to a target region encoding module M stride1 Convolution operation is performed on T' to obtain F T .

[0063] Step 233: let the feature map F obj <- T' + F T .

[0064] Further, the step 25 comprises the following steps:

[0065] Step 251: performing linear fusion processing on F obj and F stem to obtain a feature F merge .

[0066] F merge = σ (α1x F obj + α2x F stem );

[0067]

[0068] wherein σ represents a Softmax function, and α1and α2represent normalized weight coefficients;

[0069] Step 252: performing residual connection on F merge and F stem to obtain F fusion .

[0070] Further, the step 3 comprises the following steps:

[0071] Step 31: processing the image I according to a target region perception subnetwork N obj to obtain a feature map G obj .

[0072] Step 32: processing the image I according to an image fine-grained feature extraction subnetwork N stem to obtain a feature map G stem .

[0073] Step 33: processing the image I according to a feature fusion subnetwork N fusionComputing the feature map G fusion ;

[0074] Step 34: processing G local1 according to the image local feature extraction subnetwork N fusion , to obtain the feature map G local1 ;

[0075] Step 35: processing G global1 according to the image global feature extraction subnetwork N fusion , to obtain the feature map G global1 ;

[0076] Step 36: processing G local2 according to the image local feature extraction subnetwork N local1 , to obtain the feature map G local2 ;

[0077] Step 37: processing G global2 according to the image global feature extraction subnetwork N local1 , to obtain the feature map G global2 ;

[0078] Step 38: processing G global3 according to the image global feature extraction subnetwork N local2 , to obtain the feature map G global3 ;

[0079] Step 39: connecting G local2 and G global1 along the channel direction, and then processing through the local feature and global feature fusion subnetwork N Merge1 , to obtain the feature map G merge1 ;

[0080] Step 310: processing G local3 according to the image local feature extraction subnetwork N merge1 , to obtain the feature map G local3 ;

[0081] Step 311: connecting G local3 and G global2 along the channel direction, and then processing through the local feature and global feature fusion subnetwork N Merge2 , to obtain the feature map G merge2 ;

[0082] Step 312: processing G local4 according to the image local feature extraction subnetwork N merge2 , to obtain the feature map G local4 ;

[0083] Step 313: connecting G local4 and Gglobal3 The connection operation is performed along the channel direction, and then the local feature and the global feature are fused through the local-global feature fusion subnetwork N Merge3 The processing is performed to obtain a feature map G merge3

[0084] Step 314: According to the channel attention module M ECA The calculation is performed to obtain a feature map G ECA

[0085] Step 315: According to the classification head module M classifier The processing is performed on G ECA to calculate the predicted probability of the image I belonging to each category;

[0086] Step 316: Among the predicted probabilities of each category, the category with the highest probability is selected as the predicted category C I of the image I, and the probability thereof is taken as the predicted probability of the image I

[0087] Step 317: C I and

[0088] Further, the step 31 includes the following steps:

[0089] Step 311: According to the target region extraction module M extraction , the calculation is performed on the image I to obtain G′;

[0090] Step 312: According to the target region encoding module M stride1 , the convolution operation is performed on G′ to obtain G T

[0091] Step 313: Let the feature map G obj ← G′ + G T .

[0092] Further, the step 33 includes the following steps:

[0093] Step 331: According to formula (2), the linear fusion processing is performed on G obj and G stem to obtain a feature G merge .

[0094] G merge = σ (α1 × G obj + α2 × G stem ); (2)

[0095] Step 332: The residual connection is performed on G merge and G stem , so as to obtain G fusion .​​​

[0096] Further, the global average pooling layer of the step 114 pools a feature map by a factor of 1, and the adaptive one-dimensional convolution layer uses a convolution kernel with a size of KxK, and the size of K is calculated according to formula (3);

[0097]

[0098] wherein C represents the number of channels of the feature map, b and γ are hyperparameters, and |·| odd represents a function of rounding to the nearest odd number.

[0099] Compared with the prior art, the present application has the following advantages:

[0100] Firstly, in the target region perception subnetwork, the present application uses adaptive contrast adjustment, maximum inter-class variance segmentation, and morphological operation to suppress the interference of low-frequency background and dynamic light, and further extracts the fine-grained convolutional coding features of apple leaf images, which is not only conducive to fully combining the morphological priori and edge fractal information of apple leaves, but also can fuse the spatial features and shallow features of the target region, so as to realize more accurate adaptive perception of the leaf region and the disease region.

[0101] Secondly, on the basis of the network shallow convolutional block, the present application introduces a lightweight Transformer block with a parallel cascade structure to extract the multi-scale texture and edge features of apple leaf images in a lightweight transverse manner, and realizes adaptive selection of the optimal feature scale, mining of long-distance dependency relationship and fusion of cross-scale information in a skip connection manner, so as to more efficiently enhance the information interaction and representation ability of the network to multi-scale features, and improve the calculation speed and robustness of the network model.

[0102] Thirdly, in the feature fusion subnetwork, the present application introduces an adaptive learning coefficient to adaptively weight the features of the two coding branches, enhances the attention degree of the network model to the feature information of the apple leaf region, and further uses a depth separable convolution and an inverted residual mechanism to further extract the rich texture detail features of the apple leaf region, so as to improve the discrimination accuracy of the network model to different diseases.

[0103] Fourthly, the present application jointly uses a weighted cross-entropy loss function and a focal loss function, which is conducive to quickly focusing the center of gravity on those difficult-to-distinguish samples, improving the tolerance ability and robustness of the loss function to the diversity of training samples, so as to solve the problem of uneven distribution of the number of disease categories of apple leaf images to a certain extent. Therefore, the present application provides an apple leaf disease classification method which can effectively resist the interference of complex background and dynamic light, and has the characteristics of high accuracy, good robustness, strong generalization ability and low calculation complexity. BRIEF DESCRIPTION OF DRAWINGS

[0104] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0105] Figure 1 is the confusion matrix obtained by the present application on the test data set.

[0106] Figure 2 is the classification prediction result given by the present application on the test images of different categories of diseases. Among them, Figure 2 (a) is the classification prediction result of black spot disease, Figure 2 (b) is the classification prediction result of black rot disease, Figure 2 (c) is the classification prediction result of cedar apple rust disease, Figure 2 (d) is the classification prediction result of healthy leaves.

[0107] Figure 3 is the flowchart of the present application. DETAILED DESCRIPTION

[0108] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should be within the scope of protection of the present application.

[0109] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0110] As Figures 1-3As shown, the present application provides an apple leaf disease classification method based on region perception and lightweight mixed Transformer network, which is carried out according to the following steps:

[0111] Step 1. Establish and initialize a deep convolutional neural network N for apple leaf disease classification apple , containing 1 target region perception subnetwork N obj ; 1 image fine-grained feature extraction subnetwork N stem ; 1 feature fusion subnetwork N fusion ; 4 image local feature extraction subnetworks, respectively denoted as N local1 , N local2 , N local3 , N local4 ; 3 image global feature extraction subnetworks, respectively denoted as N global1 , N global2 , N global3 ; 3 local feature and global feature fusion subnetworks, respectively denoted as N Merge1 , N Merge2 and N Merge3 ; 1 channel attention module M ECA ; 1 classification head module M classifier ;

[0112] Step 1.1. Establish and initialize the target region perception subnetwork N obj , containing 1 target region extraction module M extraction and 1 target region encoding module M stride1 ;

[0113] The M extraction contains 1 adaptive contrast enhancement module M conenc , 1 image segmentation module M based on maximum inter-class variance seg , 1 morphological closing operation module M based on an elliptical structural element with a size of (10, 10) morph ;

[0114] The M stride1 contains 3 groups of convolution operations, respectively Among them, contains 1 group of 18 convolution kernels with a size of 1x1, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; contains 1 group of 18 convolution kernels with a size of 3x3 and a group number of 18, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; contains 1 group of 32 convolution kernels with a size of 1x1, and 1 layer of BatchNorm2d layer;

[0115] Step 1.2 establishes and initializes the image fine-grained feature extraction subnetwork N stem , containing 1 group of convolutional layers Conv stem , 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer.

[0116] The Conv stem contains 32 convolutional kernels with a size of 3x3, and each convolutional kernel performs convolution operation with a step size of 1 pixel.

[0117] Step 1.3 establishes and initializes the feature fusion subnetwork N fusion , containing 2 learnable weight coefficients w1 and w2.

[0118] Step 1.4 establishes and initializes the image local feature extraction subnetwork N local1 , containing 3 groups of convolutional operations, respectively

[0119] The contains 1 group of 192 convolutional kernels with a size of 1x1; contains 1 group of 192 convolutional kernels with a size of 3x3 and a group number of 192, and each convolutional kernel performs convolution operation with a step size of 2 pixels, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer. contains 1 group of 64 convolutional kernels with a size of 1x1 and 1 layer of BatchNorm2d layer.

[0120] Step 1.5 establishes and initializes the image global feature extraction subnetwork N global1 , containing 1 group of pooling operations and 1 self-attention module

[0121] The contains 1 layer of average pooling operation with a size of 2x2 and a step size of 2, and 1 layer of maximum pooling operation with a size of 2x2 and a step size of 2.

[0122] Step 1.6 establishes and initializes the image local feature extraction subnetwork N local2 , containing 3 groups of convolutional operations, respectively

[0123] The contains 1 group of 384 convolutional kernels with a size of 1x1; contains 1 group of 384 convolutional kernels with a size of 3x3 and a group number of 384, and each convolutional kernel performs convolution operation with a step size of 2 pixels, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer. contains 1 group of 128 convolution kernels with a size of 1x1, and 1 BatchNorm2d layer;

[0124] Step 1.7 establishes and initializes the image global feature extraction subnetwork N global2 , containing 1 group of pooling operations and 1 self-attention module

[0125] The contains 1 layer of average pooling operation with a size of 2x2 and a step of 2, and 1 layer of maximum pooling operation with a size of 2x2 and a step of 2;

[0126] Step 1.8 establishes and initializes the image global feature extraction subnetwork N global3 , containing 1 group of pooling operations and 1 self-attention module

[0127] The contains 1 layer of average pooling operation with a size of 2x2 and a step of 2, and 1 layer of maximum pooling operation with a size of 2x2 and a step of 2;

[0128] Step 1.9 establishes and initializes the local feature and global feature fusion subnetwork N Merge1 , containing 1 group of 128 convolution kernels with a size of 1x1, and each convolution kernel performs convolution operation with a step of 1 pixel;

[0129] Step 1.10 establishes and initializes the image local feature extraction subnetwork N local3 , containing 3 groups of convolution operations, respectively

[0130] The contains 1 group of 960 convolution kernels with a size of 1x1; contains 1 group of 960 convolution kernels with a size of 3x3 and a group number of 960, and each convolution kernel performs convolution operation with a step of 2 pixels, 1 BatchNorm2d data normalization layer, and 1 ReLU6 activation function layer; contains 1 group of 128 convolution kernels with a size of 1x1, and 1 BatchNorm2d layer;

[0131] Step 1.11 establishes and initializes the local feature and global feature fusion subnetwork N Merge2 , containing 1 group of 256 convolution kernels with a size of 1x1, and each convolution kernel performs convolution operation with a step of 1 pixel;

[0132] Step 1.12 establishes and initializes the image local feature extraction subnetwork N local4 , containing 3 groups of convolution operations, respectively

[0133] The contains 1920 convolution kernels with a size of 1x1 in one group; contains 1920 convolution kernels with a size of 3x3 and a group number of 1920, and each convolution kernel performs convolution operation with a step size of 2 pixels, 1 BatchNorm2d data normalization layer, and 1 ReLU6 activation function layer; contains 256 convolution kernels with a size of 1x1 in one group, and 1 BatchNorm2d layer;

[0134] Step 1.13 establishes and initializes a local feature and global feature fusion subnetwork N Merge3 contains 960 convolution kernels with a size of 1x1 in one group, and each convolution kernel performs convolution operation with a step size of 1 pixel;

[0135] Step 1.14 establishes and initializes a channel attention module M ECA contains 1 global average pooling layer, 1 adaptive one-dimensional convolution layer, 1 Sigmoid activation function layer, and 1 matrix Hadamard product layer;

[0136] The global average pooling layer pools a feature map into one factor; the adaptive one-dimensional convolution layer uses a convolution kernel with a size of KxK, and the size of K is calculated according to formula (1);

[0137]

[0138] C represents the number of channels of the feature map, b and γ are hyperparameters, and |·| odd represents a function of rounding to the nearest odd number;

[0139] Step 1.15 establishes and initializes a classification head module M classifier contains 1 global average pooling layer and 1 classifier;

[0140] The global average pooling layer pools a feature map into one factor; the classifier contains 1 Hardwish activation function layer, 1 Dropout layer with a probability of 0.3, and 1 fully connected layer with an input unit of 960 and an output unit of 4;

[0141] Step 2. Input the training set of apple leaf disease images and the manually labeled label set C label , train the deep convolutional neural network N apple , let the training set have N testImg images in total, and let the iteration number iter←1 and the counter ind←1;

[0142] Step 2.1 Take out the ind-th image T from the training set ind and its corresponding labeled label

[0143] Step 2.2 Perform a horizontal flip operation with a probability of 0.4 and a 20° random rotation operation on the image T ind , so as to realize data enhancement of the image T ind ;

[0144] Step 2.3 Process the image T obj using the target region perception subnetwork N ind to obtain a feature map F obj ;

[0145] Step 2.3.1 Calculate the image T extraction using the target region extraction module M ind to obtain T';

[0146] Step 2.3.2 Convolve T' using the target region encoding module M stride1 to obtain F T ;

[0147] Step 2.3.3 Let the feature map F obj ← T' + F T ;

[0148] Step 2.4 Process the image T using the image fine-grained feature extraction subnetwork N stem to obtain a feature map F stem ;

[0149] Step 2.5 Calculate the feature map F fusion using the feature fusion subnetwork N fusion ;

[0150] Step 2.5.1 Perform linear fusion processing on F obj and F stem according to formulas (2)-(4) to obtain a feature F merge ;

[0151] F merge = σ (α1×F obj + α2×F stem ) (2)

[0152]

[0153] The σ represents a Softmax function, and α1 and α2 represent normalized weight coefficients;

[0154] Step 2.5.2 Obtain F merge and F stemResidual connection is performed to calculate F fusion ;

[0155] Step 2.6, F is processed by using the image local feature extraction subnetwork N local1 to obtain the feature map F fusion ; local1 ;

[0156] Step 2.7, F is processed by using the image global feature extraction subnetwork N global1 to obtain the feature map F fusion ; global1 ;

[0157] Step 2.8, F is processed by using the image local feature extraction subnetwork N local2 to obtain the feature map F local1 ; local2 ;

[0158] Step 2.9, F is processed by using the image global feature extraction subnetwork N global2 to obtain the feature map F local1 ; global2 ;

[0159] Step 2.10, F is processed by using the image global feature extraction subnetwork N global3 to obtain the feature map F local2 ; global3 ;

[0160] Step 2.11, F local2 and F global1 are connected along the channel direction, and then processed by using the local feature and global feature fusion subnetwork N Merge1 to obtain the feature map F merge1 ;

[0161] Step 2.12, F is processed by using the image local feature extraction subnetwork N local3 to obtain the feature map F merge1 ; local3 ;

[0162] Step 2.13, F local3 and F global2 are connected along the channel direction, and then processed by using the local feature and global feature fusion subnetwork N Merge2 to obtain the feature map F merge2 ;

[0163] Step 2.14, F is processed by using the image local feature extraction subnetwork N local4 to obtain the feature map F merge2 ; local4 ;

[0164] Step 2.15, Flocal4 and F global3 Along the channel direction, the connection operation is performed, and then the local feature and the global feature fusion sub-network N Merge3 is processed to obtain the feature map F merge3 ;

[0165] Step 2.16 utilizes the channel attention module M ECA to calculate the feature map F merge3 according to formula (5), to obtain the feature map F ECA ;

[0166]

[0167] The GAP(·) represents a global average pooling operation, Conv 1×1 (·) represents a one-dimensional convolution operation, δ(·) represents a Sigmoid function, and represents a matrix Hadamard product.

[0168] Step 2.17 utilizes the classification head module M classifier to process F ECA to calculate the predicted probability of the image T ind belonging to each class.

[0169] Step 2.18 selects the class with the highest probability among the predicted probabilities of each class as the predicted class of the image T ind , and takes the probability as the predicted probability of the image T ind .

[0170] Step 2.19 sets ind←ind+1, if ind>N testImg , then go to step 2.20, otherwise return to step 2.1.

[0171] Step 2.20 calculates the predicted loss value L total of the training set according to formula (6)-(9).

[0172]

[0173]

[0174] L total = ζ1×L crossEntropy + ζ2×L focal (9)

[0175] The L focal represents an average focal loss value, L crossEntropy represents an average cross-entropy loss value, α j and ξ are hyperparameters, and k represents the number of disease types contained in the training set.ij represents the probability that the i-th image in the training set belongs to the j-th disease class, y i represents the true label of the i-th image in the training set, represents the weight coefficient of the inverse frequency of the y i -th disease class, represents the probability that the i-th image in the training set belongs to the y i -th disease class, N j represents the number of images in the training set belonging to the j-th disease class, yi represents the number of images in the training set belonging to the y i -th disease class, ζ1and ζ2represent preset weight coefficients, in this embodiment, ζ1= 0.5 and ζ2= 0.5;

[0176] Step 2.21 sets iter <- iter + 1, if the iteration number iter > Total_iter, a trained deep convolutional neural network N apple is obtained, and step 3 is entered, otherwise, the parameters of N total are updated by using the back-propagation algorithm based on the stochastic gradient descent method and the prediction loss L apple , a counter ind is set to 1, and step 2.1 is entered to reprocess all images in the training set, wherein Total_iter represents a preset iteration number, in this embodiment, Total_iter = 300;

[0177] Step 3. inputs an apple leaf disease image I to be processed, the color space of which is RGB, and uses the trained deep convolutional neural network N apple to predict the disease class to which I belongs.

[0178] Step 3.1 processes the image I by using the target region perception subnetwork N obj , to obtain a feature map G obj .

[0179] Step 3.1.1 calculates the image I by using the target region extraction module M extraction , to obtain G';

[0180] Step 3.1.2 convolves G' by using the target region encoding module M stride1 , to obtain G T .

[0181] Step 3.1.3 sets the feature map G obj <- G' + G T .

[0182] Step 3.2 processes the image I by using the image fine-grained feature extraction subnetwork N stem , to obtain a feature map Gstem ;

[0183] Step 3.3. The feature fusion subnetwork N fusion is calculated fusion ;

[0184] Step 3.3.1. According to formula (10), G obj and G stem are linearly fused to obtain the feature G merge ;

[0185] G merge = σ (α1×G obj + α2×G stem ) (10)

[0186] Step 3.3.2. G merge and G stem are connected in residual connection, thereby calculating G fusion ;

[0187] Step 3.4. The image local feature extraction subnetwork N local1 processes G fusion to obtain the feature map G local1 ;

[0188] Step 3.5. The image global feature extraction subnetwork N global1 processes G fusion to obtain the feature map G global1 ;

[0189] Step 3.6. The image local feature extraction subnetwork N local2 processes G local1 to obtain the feature map G local2 ;

[0190] Step 3.7. The image global feature extraction subnetwork N global2 processes G local1 to obtain the feature map G global2 ;

[0191] Step 3.8. The image global feature extraction subnetwork N global3 processes G local2 to obtain the feature map G global3 ;

[0192] Step 3.9. G local2 and G global1 are connected along the channel direction, and then processed by the local feature and global feature fusion subnetwork N Merge1 to obtain the feature map G merge1 ;

[0193] Step 3.10 Utilize the image local feature extraction subnetwork N local3 Processing G merge1 The feature map G is obtained. local3 ;

[0194] Step 3.11 G local3 and G global2 The connection operation is performed along the channel direction, followed by a subnetwork N that fuses local and global features. Merge2 The feature map G is obtained through processing. merge2 ;

[0195] Step 3.12 Utilize the image local feature extraction subnetwork N local4 Processing G merge2 The feature map G is obtained. local4 ;

[0196] Step 3.13 will G local4 and G global3 The connection operation is performed along the channel direction, followed by a subnetwork N that fuses local and global features. Merge3 The feature map G is obtained through processing. merge3 ;

[0197] Step 3.14 Utilize the channel attention module M ECA The calculation is performed to obtain the feature map G. ECA ;

[0198] Step 3.15 Utilize the classification head module M classifier For G ECA The process involves calculating the predicted probability that image I belongs to each category.

[0199] Step 3.16: Among the predicted probabilities of each category, select the category with the highest probability as the predicted category C of image I. I And use its probability as the predicted probability of image I.

[0200] Step 3.17 Output C I and

[0201] Example:

[0202] To verify the effectiveness of the present application, the PlantPathology Apple Dataset dataset is selected for experiment, and its download address is https: / / www.kaggle.com / lextoumbourou / plantvillageapplecolor. The dataset contains a total of 3171 apple leaf disease images, involving apple black spot, black rot, cedar apple rust and healthy leaves in four categories. In the experiment, 80% of the images are randomly selected as the training set, and the remaining 20% of the images are used as the test set.

[0203] Figure 1 The confusion matrix obtained by the present application on the test dataset is shown. Figure 2 The classification prediction results given by the present application on the test images of different categories of diseases are shown, wherein, Figure 2 (a) is the classification prediction result of black spot, Figure 2 (b) is the classification prediction result of black rot, Figure 2 (c) is the classification prediction result of cedar apple rust, Figure 2 (d) is the classification prediction result of healthy leaves. It can be seen that the present application realizes accurate classification of apple black spot, black rot, cedar apple rust and healthy leaves.

[0204] Meanwhile, Table 1 shows the classification performance comparison of the method of the present application with AlexNet, ResNet34, GoogleNet, VGG16, Alex+Inception, AICNN, and AMMFNet, and the objective evaluation indexes include accuracy, precision, sensitivity, specificity, and F1-score. As can be seen from Table 1, the method of the present application has achieved the highest evaluation score in all five objective indexes. Among them, due to the insufficient adaptive selection capability of AlexNet and VGG16 for the best multi-scale features, the classification accuracy of apple leaf disease of the two is relatively low; ResNet34 increases the skip connection between different scales, so that ResNet34 can extract and nonlinearly fuse the best multi-scale features, and its classification accuracy is improved compared with AlexNet and VGG16; GoogleNet and Alex+Inception both use convolution kernels with parallel cascade structure, which can simultaneously perform convolution on multiple scales, thereby extracting independent features belonging to different scales in the same image at one time, and then converging the features with strong correlation, thereby achieving better classification performance than ResNet34; AMMFNet introduces a channel attention mechanism, which helps to explore the complementarity between RGB and multispectral image data, and further improves the classification accuracy; AICNN introduces a transfer learning strategy based on the VGG16 model, and jointly uses a global average pooling layer, a batch normalization layer, and a full connection layer, and its objective evaluation result reaches more than 99%; the present application effectively integrates regional perception coding and backbone network coding, enhances the attention degree of leaf region feature information, and sends the feature information with enhanced regional attention degree into a convolution block and a Transformer block, further extracts local feature information and global feature information of the target region, and introduces an attention mechanism in the deep layer of the network to give a higher weight coefficient to the key feature map, thereby achieving the highest classification efficiency. At the same time, the network parameter amount of the method of the present application is only 4.14% of that of AlexNet, 10.65% of that of ResNet34, 41.82% of that of GoogleNet, 1.80% of that of VGG16, 42.59% of that of Alex+Inception, 1.19% of that of AMMFNet, and 16.43% of that of AICNN. Obviously, the network model volume and calculation amount of the method of the present application are significantly lower than those of the other several methods.

[0205] In summary, the present application provides an apple leaf disease classification method which can effectively resist complex background and dynamic light interference, and has the characteristics of high accuracy, good robustness, strong generalization ability, and low calculation complexity.

[0206] Table 1 Comparison of classification results of different apple leaf disease classification methods

[0207]

[0208] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0209] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0210] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0211] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0212] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0213] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the whole or part of the technical solutions which essentially contribute to the prior art can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The above-mentioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0214] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An apple leaf disease classification method of regional perception and lightweight hybrid Transformer network, characterized in that, The method comprises the following steps: Step 1: Establish and initialize a deep convolutional neural network N for apple leaf disease classification apple ; the deep convolutional neural network N apple comprises: a target region perception subnetwork N obj , an image fine-grained feature extraction subnetwork N stem , a feature fusion subnetwork N fusion , four image local feature extraction subnetworks, respectively denoted as N local1 , N local2 , N local3 , N local4 , three image global feature extraction subnetworks, respectively denoted as N global1 , N global2 , N global3 , three local feature and global feature fusion subnetworks, respectively denoted as N Merge1 , N Merge2 and N Merge3 , a channel attention module M ECA and a classification head module M classifier ; Step 2: input a training set of apple leaf disease image and a manually labeled label set C label , and train the deep convolutional neural network N apple , to obtain the trained deep convolutional neural network N apple ; the training set has N testImg images, and let the iteration number iter ind 1. Step 3: input an apple leaf disease image I to be processed, the color space of the apple leaf disease image I to be processed being RGB; acquire the trained deep convolutional neural network N according to the step 2 apple The disease category of the apple leaf disease image I to be processed is predicted.

2. The method of apple leaf disease classification using region-aware and lightweight hybrid Transformer network according to claim 1, wherein The step 1 comprises the following steps: Step 11: Establishing the target region perception sub-network N obj and initializing it; the target region perception sub-network N obj contains: 1 target region extraction module M extraction and 1 target region encoding module M stride1 ; the M extraction includes: 1 adaptive contrast enhancement module M conenc , 1 maximum inter-class variance based image segmentation module M seg , and 1 morphological closing operation module M morph based on an elliptical structural element with a size of (10, 10); The M stride1 contains 3 groups of convolution operations, respectively wherein, contains 1 group of 18 convolution kernels with a size of 1x1, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; contains 1 group of 18 convolution kernels with a size of 3x3 and a group number of 18, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; contains 1 group of 32 convolution kernels with a size of 1x1, 1 layer of BatchNorm2d layer; Step 12: establish an image fine-grained feature extraction subnetwork N stem and initialize it; the image fine-grained feature extraction subnetwork N stem includes a convolutional layer Conv stem , a BatchNorm2d data normalization layer and a ReLU6 activation function layer; the convolutional layer Conv stem includes 32 convolutional kernels with a size of 3*3, and each convolutional kernel performs convolution operation with a step size of 1 pixel; Step 13: Establishing a feature fusion sub-network N fusion and initializing it; the feature fusion sub-network N fusion includes 2 learnable weight coefficients w1 and w2; Step 14: Establishing an image local feature extraction subnetwork N local1 and initializing it; the image local feature extraction subnetwork N local1 includes three groups of convolution operations; the convolution operations are respectively The contains one group of 192 convolution kernels with a size of 1x1; contains one group of 192 convolution kernels with a size of 3x3 and a group number of 192, and each convolution kernel performs convolution operation with a step of 2 pixels, one BatchNorm2d data normalization layer, and one ReLU6 activation function layer; contains one group of 64 convolution kernels with a size of 1x1 and one BatchNorm2d layer; Step 15: Establishing an image global feature extraction subnetwork N global1 and initializing it; the image global feature extraction subnetwork N global1 contains 1 set of pooling operations and 1 self-attention module The pooling operation contains 1 layer of average pooling operation with a size of 2x2 and a step of 2, and 1 layer of max pooling operation with a size of 2x2 and a step of 2; Step 16: Establishing an image local feature extraction subnetwork N local2 and initializing it; the image local feature extraction subnetwork N local2 contains 3 groups of convolution operations, respectively The contains 1 group of 384 convolution kernels with a size of 1x1; contains 1 group of 384 convolution kernels with a size of 3x3 and a group number of 384, and each convolution kernel performs convolution operation with a step size of 2 pixels, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; contains 1 group of 128 convolution kernels with a size of 1x1 and 1 layer of BatchNorm2d layer; Step 17: establishing an image global feature extraction subnetwork N global2 and initializing it; the image global feature extraction subnetwork N global2 contains 1 set of pooling operations and 1 self-attention module The contains 1 layer of average pooling operation with a size of 2x2 and a step of 2 and 1 layer of max pooling operation with a size of 2x2 and a step of 2; Step 18: establishing an image global feature extraction subnetwork N global3 and initializing it; the image global feature extraction subnetwork N global3 contains 1 set of pooling operations and 1 self-attention module The contains 1 layer of average pooling operation with a size of 2x2 and a step of 2 and 1 layer of maximum pooling operation with a size of 2x2 and a step of 2; Step 19: Establishing a local feature and global feature fusion sub-network N Merge1 and initializing the same; the local feature and global feature fusion sub-network N Merge1 contains 1 group of 128 convolution kernels with a size of 1x1, and each convolution kernel performs convolution operation with a step size of 1 pixel; Step 110: Establishing an image local feature extraction subnetwork N local3 and initializing it; the image local feature extraction subnetwork N local3 contains three groups of convolution operations, respectively The contains a group of 960 convolution kernels with a size of 1x1; contains a group of 960 convolution kernels with a size of 3x3 and a group number of 960, and each convolution kernel performs convolution operation with a step of 2 pixels, 1 layer of BatchNorm2d data normalization layer, and 1 layer of ReLU6 activation function layer; contains a group of 128 convolution kernels with a size of 1x1 and 1 layer of BatchNorm2d layer; Step 111: a local feature and global feature fusion sub-network N is established Merge2 and is initialized; the local feature and global feature fusion sub-network N Merge2 contains 1 group of 256 convolution kernels with a size of 1x1, and each convolution kernel performs convolution operation with a step size of 1 pixel; Step 112: Establishing an image local feature extraction subnetwork N local4 and initializing it; the image local feature extraction subnetwork N local4 contains three groups of convolution operations, respectively The contains one group of 1920 convolution kernels with a size of 1x1; contains one group of 1920 convolution kernels with a size of 3x3 and a group number of 1920, and each convolution kernel performs convolution operation with a step of 2 pixels, 1 layer of BatchNorm2d data normalization layer, and 1 ReLU6 activation function layer; contains one group of 256 convolution kernels with a size of 1x1 and one BatchNorm2d layer; Step 113: a local feature and global feature fusion sub-network N is established Merge3 and is initialized; the local feature and global feature fusion sub-network N Merge3 contains 1 group of 960 convolution kernels with a size of 1x1, and each convolution kernel performs convolution operation with a step size of 1 pixel; Step 114: establishing a channel attention module M ECA and initializing it; the channel attention module M ECA comprising a global average pooling layer, an adaptive one-dimensional convolution layer, a Sigmoid activation function layer, and a matrix Hadamard product layer. Step 115: Establishing a classification head module M classifier and initializing it; the classification head module M classifier contains a global average pooling layer and a classifier; The global average pooling layer of the step 114 pools a feature map into 1 factor; 3. The method of apple leaf disease classification using region-aware and lightweight hybrid Transformer network according to claim 1, wherein, The step 2 comprises the following steps: Step 21 : Take the indth image T from the training set ind and its corresponding labeled label Step 22: Apply horizontal flip operation with probability 0.4 and 20° random rotation operation to image T ind to achieve data augmentation of image T ind ; Step 23: According to the target region perception sub-network N obj Process the image T ind , to obtain the feature map F obj ; Step 24: according to the image fine-grained feature extraction sub-network N stem processing the image T ind , to obtain a feature map F stem ; Step 25: According to the feature fusion sub-network N fusion Computing the feature map F fusion ; Step 26: According to the image local feature extraction sub-network N local1 Process F fusion , to obtain the feature map F local1 ; Step 27: According to the image global feature extraction sub-network N global1 Process F fusion , to obtain the feature map F global1 ; Step 28: According to the image local feature extraction sub-network N local2 Process F local1 , to obtain the feature map F local2 ; Step 29: According to the image global feature extraction subnetwork N global2 Process F local1 , to obtain the feature map F global2 ; Step 210: extracting the global feature of the image according to the image global feature extraction subnetwork N global3 Process F local2 , to obtain a feature map F global3 ; Step 211: F local2 and F global1 The connection operation is performed along the channel direction, and then the local feature and global feature fusion subnetwork N Merge1 is used for processing to obtain the feature map F merge1 ; Step 212: extracting local features of the image according to the image local feature extraction subnetwork N local3 Process F merge1 , to obtain a feature map F local3 ; Step 213: F local3 and F global2 Connect along the channel direction, and then fuse local features and global features through the subnetwork N Merge2 Process to obtain the feature map F merge2 ; Step 214: extracting local features of the image according to the image local feature extraction subnetwork N local4 Process F merge2 , to obtain a feature map F local4 ; Step 215: F local4 and F global3 The connection operation is performed along the channel direction, and then the local feature and global feature fusion subnetwork N Merge3 is used for processing to obtain the feature map F merge3 ; Step 216: According to the channel attention module M ECA , the feature map F merge3 is calculated according to formula (1), and the feature map F ECA is obtained. where GAP(·) denotes a global average pooling operation, Conv 1×1 (·) denotes a one-dimensional convolution operation, and δ(·) denotes a Sigmoid function, denotes a matrix Hadamard product. Step 217: Utilizing the classification head module M classifier F ECA processing, calculating the predicted probabilities of the image T ind belonging to each class; Step 218: In the predicted probabilities of various categories, the category with the highest probability is selected as the predicted category of image T ind , and its probability is taken as the predicted probability of image T ind . Step 219: Let ind <- ind + 1, if ind > N testImg then go to step 220, otherwise return to step 21; Step 220: Calculate the prediction loss value L of the training set total : L total = ζ1 x L crossEntropy + ζ2 x L focal ; The L focal represents the average focal loss value, L crossEntropy represents the average cross-entropy loss value, a j and ξ are hyperparameters, k represents the number of disease classes contained in the training set, p ij represents the probability that the model predicts the i-th image of the training set belongs to the j-th disease class, y i represents the true label of the i-th image of the training set, W yi represents the weight coefficient of the inverse frequency of the y i disease class, p yi represents the probability that the model predicts the i-th image of the training set belongs to the y i disease class, N j represents the number of images belonging to the j-th disease class in the training set, N yi represents the number of images belonging to the y i disease class in the training set, ζ1 and ζ2 represent preset weight coefficients; Step 221: let iter <- iter + 1, if the iteration number iter > Total_iter, then the trained deep convolutional neural network N is obtained apple , turn to step 3, otherwise, update the parameters of N total using the back-propagation algorithm based on the stochastic gradient descent method and the prediction loss L apple , let the counter ind <- 1, turn to step 21 to reprocess all images in the training set, and Total_iter represents the preset iteration number.

4. The method of claim 3, wherein the method is characterized by, The step 23 further comprises the following steps: Step 231 : according to the target region extraction module M extraction On the image T ind Calculate, get T'; Step 232: encode the target region according to the target region encoding module M stride1 F = T' * G T ; Step 233: let the feature map F obj ← T' + F T .

5. The method of apple leaf disease classification using region-aware and lightweight hybrid Transformer network according to claim 3, wherein, The step 25 comprises the following steps: Step 251: F obj and F stem linear fusion processing to obtain feature F merge ; F merge = σ (α1 x F obj + α2 x F stem ); Wherein, σ represents a Softmax function, and α1 and α2 represent normalized weight coefficients; Step 252: F merge and F stem performing residual connection, thereby calculating F fusion .

6. The method of apple leaf disease classification using region-aware and lightweight hybrid Transformer network according to claim 1, wherein, The step 3 comprises the following steps: Step 31: According to the target region perception sub-network N obj Processing the image I to obtain the feature map G obj ; Step 32: extracting image fine-grained features according to the image fine-grained feature extraction subnetwork N stem processing the image I to obtain the feature map G stem ; Step 33: According to the feature fusion sub-network N fusion Computing the feature map G fusion ; Step 34: extracting local features from the image according to the image local feature extraction subnetwork N local1 Process G fusion , to obtain a feature map G local1 ; Step 35: extracting the global feature of the image according to the image global feature extraction subnetwork N global1 Process G fusion , to obtain a feature map G global1 ; Step 36: extracting local features from the image according to the image local feature extraction subnetwork N local2 Process G local1 , to obtain a feature map G local2 ; Step 37: extracting the global feature of the image according to the image global feature extraction subnetwork N global2 Process G local1 , to obtain a feature map G global2 ; Step 38: extracting the global feature of the image according to the image global feature extraction subnetwork N global3 Process G local2 , to obtain a feature map G global3 ; Step 39: connect G local2 and G global1 along the channel direction, and then pass through the local feature and global feature fusion subnetwork N Merge1 for processing to obtain the feature map G merge1 ; Step 310: extracting local features of the image according to the image local feature extraction subnetwork N local3 Process G merge1 , to obtain a feature map G local3 ; Step 311: G local3 and G global2 The connection operation is performed along the channel direction, and then the local feature and global feature fusion subnetwork N Merge2 is processed to obtain the feature map G merge2 ; Step 312: extracting local features of the image according to the image local feature extraction subnetwork N local4 Process G merge2 , to obtain a feature map G local4 ; Step 313: G local4 and G global3 Connect along the channel direction, and then fuse local features and global features through the subnetwork N Merge3 Process to obtain the feature map G merge3 ; Step 314: According to the channel attention module M ECA The calculation is performed to obtain the feature map G ECA ; Step 315: according to the classification head module M classifier G ECA processing, the predicted probability of the image I belonging to each category is calculated; Step 316: Among the predicted probabilities of the respective categories, the category with the highest probability is selected as the predicted category C of the image I I and its probability is taken as the predicted probability of the image I Step 317: output C I and 7. The method of apple leaf disease classification using region-aware and lightweight hybrid Transformer network according to claim 6, wherein, The step 31 comprises the following steps: Step 311 : according to the target region extraction module M extraction G' is computed on the image I; Step 312: encode the target region according to the target region encoding module M stride1 G' = G * G T ; Step 313: Let the feature map G obj ← G' + G T .

8. The method of apple leaf disease classification using region-aware and lightweight hybrid Transformer network according to claim 6, wherein, The step 33 comprises the following step: step 331, performing linear fusion processing on G obj and G stem according to formula (2) to obtain a feature G merge ; G merge = σ (α1x G obj + α2x G stem ); (2) Step 332 performs a residual connection to compute G merge and G stem perform a residual connection to compute G fusion ; 9. The method of claim 2, wherein the region-aware and lightweight hybrid Transformer network is a ResNet50 network. The global average pooling layer of the step 114 pools a feature map into 1 factor; The adaptive one-dimensional convolution layer uses a convolution kernel with a size of KxK, and the size of K is calculated according to formula (3); where C denotes the number of channels of the feature map, b and g are hyperparameters, and | · | denotes the absolute value. odd denotes the function that rounds to the nearest odd integer.

Images