A grape leaf disease spot image segmentation method based on a transformer

Abstract

The present application belongs to the field of agricultural information technology, and particularly relates to a grape leaf disease spot image segmentation method based on a Transformer, wherein the present application proposes a grape leaf disease spot image segmentation method based on a Transformer, namely LRT, aiming at grape leaf disease spot features, which is designed to explore reversible attention mechanisms from the perspective of wavelet transform and CNN to solve the problem of context semantic information loss of the Transformer caused by traditional down-sampling, and the present application realizes self-attention learning by performing lossless down-sampling on keys-value, so as to help to pursue a better trade-off between efficiency and accuracy and improve the segmentation performance of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of agricultural information technology, and particularly relates to a grape leaf disease spot image segmentation method based on a Transformer. BACKGROUND

[0002] Plant diseases are a major threat to the agricultural production chain and product quality. In the agricultural community, grape black measles is considered one of the most widely studied and most destructive fungal diseases, and at the same time, black measles can spread fungi throughout the plantation and cause epidemics. Identifying images of grape leaves is an effective way to detect diseases, and in particular, accurate segmentation of images can improve the accuracy of identification, so as to formulate targeted prevention and control strategies, minimize treatment costs, and increase the yield and economic benefits of plants. There are usually the following segmentation methods:

[0003] (1) Traditional segmentation methods

[0004] Traditional segmentation methods are a basic task in computer vision, and their essence is to classify pixels in an image with labels. Traditional segmentation methods manually extract low-level features (gray, color, texture) as a basic layer, and then researchers classify pixels at the pixel level through segmentation methods. For example, threshold-based methods (Otsu), edge methods, region growing methods, and clustering methods. Traditional segmentation methods require a large amount of prior knowledge, and for data sets with a small number of samples, it is difficult to achieve satisfactory segmentation results. In addition, traditional segmentation methods rely on manual feature extraction, which greatly increases the work intensity of segmentation.

[0005] (2) Deep learning network

[0006] An effective solution is to use deep learning networks to accurately segment diseases. Due to the powerful representation learning ability of deep learning networks, people's interest in introducing them into the field of image segmentation has increased, and in addition, they have good generalization ability and do not require prior parameters. Classic deep learning network segmentation models include FCN, Unet, PSPNet, DeeplabV3, and DeeplabV3+, which have achieved satisfactory segmentation results on public data sets. At the same time, semantic segmentation models based on deep learning (especially CNN-based networks) are widely used in plant disease assessment research to improve the accuracy and robustness of segmentation. However, the above deep learning network models ignore the comprehensive consideration of background and spatial feature information, and cannot capture long-distance dependencies. In addition, areas in the background that have similar colors and textures to disease spots are easily mistaken by the model as a basis for judgment, which often misleads the model, leading to poor recognition performance.

[0007] (3)Transformer

[0008] Recently, the Transformer has achieved superior performance over existing convolutional neural networks in segmentation tasks, which relies on multi-head self-attention mechanism to capture long-range dependencies. However, the sequence length of high-resolution image patches is much longer than that of NLP tokens (words), which results in high time and space complexity in computing multi-head self-attention, and the computational cost and memory consumption are quadratic O(N 2 C) with the resolution of the input image. To solve the problem of low efficiency of the Transformer caused by high computation, a large number of Transformer variants for segmentation tasks have emerged. Some researchers reduce the computational cost of self-attention and improve the efficiency of the model by expanding the receptive field. Although the self-attention mechanism of the Transformer can well simulate the global interaction between tokens, it lacks attention mechanisms for exchanging information within local regions. Therefore, some works attempt to introduce convolutional operations into visual Transformers. A large number of experiments have proved that the effective combination of the advantages of convolutional neural networks in extracting low-level features and strengthening locality and the advantages of Transformers in establishing long-range dependencies can obtain higher performance and better convergence. On the other hand, some researchers reduce the computational cost from the perspective of key-value pair downsampling. However, these down-samplings are irreversible and will cause the loss of context semantic information, thereby reducing the performance of the model for the segmentation task. In fact, researchers have proved that irreversible downsampling will cause the loss of high-frequency semantic information and will harm the translational invariance of deep networks. In addition, the Transformer is difficult to train due to the excessive amount of parameters, and a large amount of data sets are needed to obtain performance comparable to convolutional neural networks (CNN), which limits the application in the scene with limited computing resources. SUMMARY

[0009] The present application is aimed at the characteristics of grape leaf spot, and proposes a grape black measles semantic segmentation method based on transformer, namely Local Reversible Transformer (LRT), which aims to explore reversible attention mechanisms from the perspective of wavelet transform and CNN to solve the problem of loss of context semantic information of the Transformer caused by traditional downsampling.

[0010] The local reversible transformer is an encoder-decoder architecture, which adopts a progressive pyramid strategy to extract semantic prior features, and is divided into four stages. Unlike the single-stage design, the input of the multi-stage design model has a higher resolution to ensure sufficient information input, and it also reduces redundancy through downsampling, thereby reducing the computational workload. The four stages of the model correspond to 4, 8, 16 and 32 times downsampling respectively, which reduces the height and width of the feature map and increases the number of channels. The dimension of the feature in the same stage remains unchanged, and after downsampling, the dimension becomes twice the original. Each stage is composed of four parts, which are overlapping token merging, local learning unit (LLU), wavelet enhanced attention module (WEA) and convolution feedforward neural network. The LLU and WEA with fine-grained semantic information are collectively referred to as the progressive refinement wavelet block (PRW) in the application. In the decoder part, a lightweight MLP is used to integrate the feature information of each stage to generate the final segmentation mask.

[0011] The application adopts a pyramid structure to extract multi-scale interaction information and obtain spatial fine-grained features from low resolution. Specifically, given an input image with a size of HxWx3, we use convolution (7x7 convolution kernel, step size 2) to generate a feature map of H / 4xW / 4. After convolution, Batch Normalization normalization and activation function are performed. In this way, in the i-th stage, the resolution of the token is

[0012] H / 2 (i+1) xW / 2 (i+1) , C i is the generated channel number, where i∈{1,2,3,4}. The model of the application takes advantage of the progressive pyramid strategy to obtain greater competitiveness with less computational cost.

[0013] Overlapping token merging. Non-overlapping tokens lose the connection between adjacent tokens to some extent, so we use overlapping token merging, and the two-dimensional reconstructed feature map of the previous stage is used as the input of the i-th stage. Then we learn a function f(·) to map the x i-1 feature to a new token f(x i ) with a channel size of C i-1 , where f(·) is a two-dimensional convolution operation with a convolution kernel size of kxk, a step size of s and a padding size of p. The height and width of the new token feature are as follows:

[0014]

[0015] Overlapping token merging adjusts the token feature dimension and the number of tokens in each stage by changing the parameters of the convolution operation. In this operation, the convolution parameters for the first stage are set to k=7, s=4, p=3, and the parameters for the remaining stages are set to k=3, s=2, p=1.

[0016] Local Learning Units (LLUs). The properties of convolution (i.e., shift, scaling, and distortion invariance) determine that it primarily captures local and topological information in an image. The inherent properties of Transformers (i.e., dynamic attention, global context, and better generalization) mainly rely on global information to construct interaction information between different tokens, but neglect local feature information. Therefore, by coupling the advantages of CNNs and Transformers, feature representation learning ability can be improved. Furthermore, some studies have shown that introducing convolution into...

[0017] The Transformer architecture can guarantee shift and distortion invariance to a certain extent. Therefore, we construct a Local Learning Unit (LLU) by integrating the advantages of CNN and Transformer to enhance local perception capabilities and extract richer semantic information. The LLU is defined as:

[0018] LLU(X i =DWConv(X) i )+X i

[0019] in DWConv(·) represents a depthwise separable convolution operation.

[0020] Wavelet Enhanced Attention Mechanism (WEA). To alleviate the quadratic increase in memory and computation required by attention mechanisms, some networks utilize pooling operations to reduce spatial dimensionality. However, this downsampling is irreversible and leads to loss of contextual semantic information. Considering that wavelet transform represents information at different scales, it allows for time-frequency domain analysis. Therefore, this invention designs an attention mechanism that represents features in a reversible manner, called Wavelet Enhanced Attention Mechanism (WEA), which introduces Discrete Wavelet Transform (DWT) to replace the previous downsampling operation in the multi-head attention mechanism. First, the reconstructed input image is decomposed into four frequency sub-bands using DWT. Then, convolution is performed to further apply spatial locality to the frequency sub-bands. Finally, multi-head attention feature learning and discrete wavelet inverse transform (IDWT) image reconstruction are performed in parallel.

[0021] Specifically, firstly, a two-dimensional reconstructed token mapping is given. As input, the token is transformed into X = XW through linear projection and flattening operations. C ,in Haar wavelet (DWT) with orthogonality and simple computation is used to obtain spatial and transform domain information. DWT decomposes the input image into 4 subbands where L denotes a low-pass filter and W denotes a high-pass filter. The filters and DWT can be represented as

[0022]

[0023] LL is the approximation image of the input image, representing the low frequency subband. LH, HL, HH represent the horizontal, vertical and diagonal features extracted from the original image, respectively.

[0024] Then, a 3x3 convolution is used to generate a local feature map. The captured feature map X c Linear projection as keys

[0025] and values The attention mechanism of the present application Attention w is:

[0026]

[0027] where, and V i w represent the keys and values corresponding to the i-th head, respectively. Finally, inverse DWT (IDWT) is implemented based on the i-th decomposed low frequency coefficient and the high frequency coefficients from 1-th to i-th. The reconstructed image retains all the high frequency and low frequency information of the original image. Each head learns the global information of the i-th stage of feature aggregation, and the specific implementation can be summarized as follows:

[0028] WE(X) = MultiHead w (XW q ,X c W k ,X c W v ,X r )

[0029]

[0030] where X r represents local context information, and W O represents a transformation matrix.

[0031] Feedforward neural network. A 3x3 depth separable convolution is added between the LayerNorm normalization layer and the activation function in the feedforward network, with a padding size of 1. Using a depth separable convolution can reduce the number of parameters and improve efficiency. ​

[0032] The application can solve the problem of loss of context semantic information of the Transformer caused by the traditional down-sampling of the self-attention mechanism, and has the advantages of wavelet transform and CNN, learns multi-granularity semantic information, improves the performance of the model, helps to pursue a better trade-off between efficiency and accuracy, and has the following beneficial effects:

[0033] A semantic segmentation dataset of grape black measles is constructed;

[0034] Unlike existing Transformers, the application realizes down-sampling in a reversible manner and can better trade off accuracy and parameters;

[0035] The local learning unit (LLU) is used to enhance the local perception ability and extract more rich spatial semantic information;

[0036] A wavelet attention mechanism (WEA) is constructed, which captures more topological structure information through lossless and reversible down-sampling of keys and values, and realizes self-attention learning;

[0037] The segmentation model achieves the best results in the field in terms of qualitative and quantitative indicators. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The model architecture diagram of the grape leaf spot image segmentation method based on the Transformer of the application is shown in the figure;

[0039] Figure 2 The WEA mechanism diagram of the application is shown in the figure;

[0040] Figure 3 The detailed design diagram of the WEA of the application is shown in the figure. DETAILED DESCRIPTION

[0041] The overall framework diagram of the grape leaf spot image segmentation method based on the Transformer (local reversible Transformer, LRT) is shown in the figure Figure 1 , which is an end-to-end model; as Figure 2 shown, the wavelet enhanced attention mechanism of the application is different from the traditional down-sampling attention; as Figure 3 shown, the detailed design diagram of the wavelet enhanced attention module of the application. In the training stage, the AdamW optimizer with weight decay of 0.0005 is used to optimize the local reversible Transformer. We use the learning rate decay "poly" LR strategy to train LRT (lr=baserx(1-epoch / maxiter) power), each parameter is set as follows: "poly" LR policy factor is set to 1, the initial learning rate is 6x10 -5 , a total of 160,000 iterations, Drop path is set to 0.1. The model is trained using grape black measles image with resolution size of 256x256.

[0042] The training data comes from Plant Village, which is a public and fair dataset for crop disease and pest identification. We use 1383 grape black measles images from it. Currently, there is no publicly available grape disease dataset for segmentation tasks. In order to meet the requirements of segmentation tasks, we established a dataset named GBMS, which uses Labelme as a labeling tool to manually label the above images to distinguish the disease area and the leaf area. The labeled data is converted to PASCAL VOC 2012 data format, which has semantic labels for foreground and background objects. In order to enrich the diversity of images and avoid data overfitting, we apply the semantic segmentation framework MMSegmentation to data augmentation, including geometric transformation (mean subtraction, random left-right flipping, random cropping, random sampling, color jittering, etc.) and color space conversion (gray, brightness, contrast, saturation and hue changes).

[0043] In order to evaluate the effectiveness of LRT, the model is compared with other segmentation methods. The evaluation indicators include: class accuracy, overlap (IoU), mean intersection over union (mIoU), pixel accuracy (PA), mean pixel accuracy (mPA) and mDice. Among them, mDice, IoU and mIoU are used to evaluate the similarity between the network prediction segmentation result and the human annotation result. At the same time, the parameters (Params) and gigaflops per second (GFLOPs) of each model are analyzed.

[0044] Table 1 shows the quantitative results of the classic segmentation model on the GBMS dataset, and the results show that the classic segmentation model performs poorly in evaluation indicators. For black measles with complex texture structure, the classic segmentation model achieves more than 90% in the IoU of leaf and background. Due to the pooling downsampling operation, PSPNet has a serious loss of context information between different sub-regions, resulting in poor segmentation effect. Unet will segment the normal area of grape leaves as a diseased area, resulting in poor segmentation results. The mIoU of Unet(resnet50) and the mPA of DeeplabV3+(xception) achieve the highest scores, respectively. This means that the segmentation results of these two models have a high similarity with the manually annotated segmentation results. However, although DeeplabV3 and DeeplabV3+ integrate more disease features, they cannot obtain the texture information of the pixel mutation part. In contrast, due to the characteristics of the attention mechanism, LRT can capture more context information, which shows that the model of the present application can achieve more significant segmentation results, and can better achieve the trade-off between params and GFLOPs.

[0045] Table 1 Comparison results of classic segmentation models on GBMS dataset

[0046]

[0047] In order to verify the robustness and accuracy of the LRT model, Table 2 shows the measurement indicator values of different Transformer architectures on background, leaves and black measles, respectively. From the table, it can be observed that LRT obtains the highest scores in IoU, PA, accuracy and Dice. For the background and leaf regions with clear contours, the present application shows higher segmentation accuracy, which means that LRT can learn more context information. The IoU result of LRT in black measles segmentation is higher than that of other comparison models, which is 3.87% higher than the highest IoU of the CSwin model, which shows that LRT has a strong ability to learn edge texture detail information. Compared with Danet and HRNet, the PA of LRT increases by 4.49% and 4.3%, respectively, which means that the segmentation results of the model of the present application have a higher similarity with the annotation results. The Transformer based on the attention mechanism can achieve good segmentation results for pixels with large gradient changes, but it is difficult to segment edge or fuzzy disease areas. In contrast, compared with other models, LRT shows better visual effects in the segmentation results of leaves, background and black measles, and improves the performance of detection and fine-grained information segmentation.

[0048] Table 2 Quantization results of different Transformer architectures on the GBMS dataset

[0049]

[0050] The comparison results of LRT and other methods in terms of params(M) and GFLOPs are shown in Tables 3 and 4. Through comparison, it can be found that LRT presents a better complexity and performance trade-off. In the classic segmentation model, DeeplabV3+(xception) has the lowest GFLOPs of 33.76, and the parameter quantity is not the lowest compared with other classic segmentation models. The classic segmentation model fails to achieve a good trade-off between GFLOPs and parameters, and neither efficiency nor performance is ideal. The GFLOPs and params values of LRT are 1.33 and 3.73M respectively, and compared with CSwin(108.92), the lightweight advantage is significant, which means that the LRT model can alleviate the problem of low computational efficiency in the semantic segmentation task. In the case of similar parameter quantity with segformer, the LRT network structure significantly improves the segmentation performance and extracts more features. In summary, the results show that the LRT segmentation result has higher similarity with the labeled result, and at the same time better realizes the trade-off between params(M) and GFLOPs.

[0051] Table 3 Comparison of classic models in terms of params(M) and GFLOPs

[0052]

[0053] Table 4 Comparison of different Transformer models in terms of params(M) and GFLOPs

[0054]

Claims

1. A method for grape leaf disease spot image segmentation based on Transformer, characterized in that, The method explores reversible attention mechanism from the perspective of wavelet transform and CNN, and constructs a local reversible Transformer model, i.e., LRT, to solve the problem of context semantic information loss caused by traditional down-sampling; The constructed LRT model adopts a progressive pyramid strategy and is divided into four stages, each of which is composed of four parts, namely, overlapping token merging, local learning unit, wavelet enhanced attention module and convolution feedforward neural network; The overlapping token merging adjusts the token feature dimension and the token number of each stage by changing the parameters of the convolution operation, reconstructs the two-dimensional feature map of the previous stage into a new token f(x As the input of the i-th stage, define a function f(·) to map x i-1 to a new token f(x i-1 with channel size C i , where f(·) is a two-dimensional convolution operation with convolution kernel size k×k, step size s, and padding size p, and the height and width of the new token feature are as follows: In this operation, the convolution parameters of the first stage are set as k=7, s=4, p=3, and the remaining stages are set as k=3, s=2, p=1; The local learning unit couples the advantages of CNN and Transformer, which can improve the feature representation learning ability, enhance the local perception ability, and extract more rich semantic information, which can be represented as: LLU(X i ) = DWConv(X i ) + X i wherein DWConv(·) denotes a depthwise separable convolution operation; The wavelet enhanced attention module designs a reversible attention mechanism to represent features, which is called wavelet enhanced attention mechanism. In the multi-head attention mechanism, the discrete wavelet transform DWT is introduced instead of the previous down-sampling operation, which is divided into three steps: Step 5.1: The input image is decomposed into four frequency subbands by DWT; The DWT decomposes the input image into 4 frequency subbands by a series of high-pass and low-pass filters where L denotes a low-pass filter and W denotes a high-pass filter. The filters and the DWT can be denoted as:​ Among them, LL is the approximation image of the input image, representing the low-frequency subband, and LH, HL, HH represent the horizontal, vertical and diagonal features extracted from the original image, respectively; Step 5.2: Convolution is performed to further impose spatial locality on the frequency subbands; A 3x3 convolution is used to generate local feature maps, X c The linear projection of the local feature maps X and the value Step 5.3: Multi-head attention feature learning and inverse discrete wavelet transform IDWT reconstruct the image in parallel operation; wherein, the attention mechanism Attention w is represented as: wherein, and denote the key and value corresponding to the i-th header, respectively, and IDWT is implemented based on the i-th decomposed low-frequency coefficient and the high-frequency coefficients from 1-th to i-th, the image reconstructed by IDWT retains all the high-frequency and low-frequency information of the original image, each header learns the global information aggregated in the i-th stage, and the formula is as follows: WE(X) = MultiHead w (XW q ,X c W k ,X c W v ,X r ) where X r represents local context information, W O represents a transformation matrix.

2. The method of claim 1, wherein the method is based on a Transformer. A 3×3 depth separable convolution with padding size 1 is added between the LayerNorm normalization layer and the activation function in the feedforward network, which reduces the number of parameters and improves efficiency.

Images