A grape leaf disease spot image segmentation method based on a cross-resolution Transformer model

Abstract

The present application belongs to the field of agricultural information technology, and particularly relates to a grape leaf disease spot segmentation method based on a cross-resolution Transformer model, which is used to solve the problem of grape leaf disease spot segmentation in a complex background in a natural field environment. Compared with the previous Transformer model based on the consideration of obtaining high-resolution features or semantic information alone, the cross-resolution Transformer model simultaneously considers the preservation of high-resolution features and strong semantic information. The cross-resolution Transformer model is an "encoder-decoder" structure, the encoder is composed of 4 stages, and i parallelly connected multi-resolution Transformer blocks are distributed in the i-th stage, arranged in a pyramid shape; the decoder is a hamburger decoder. Each Transformer block includes a large core attention mechanism, a multi-path feedforward network and a cross-resolution fusion strategy. Finally, the feature information of different scale resolutions is fused through the hamburger decoder.

Description

Technical Field

[0001] This invention belongs to the field of agricultural information technology, and specifically relates to a method for segmenting grape leaf lesion images based on a cross-resolution Transformer model. Background Technology

[0002] Leaf spots have become a major obstacle to the development of grape agriculture, directly leading to a decline in yield and quality. Monitoring disease information and formulating appropriate measures in the early stages of disease can effectively control agricultural economic losses. Automated segmentation is an important basis for plant disease detection and identification; therefore, automated segmentation of grape leaf spots helps prevent the spread of disease. However, the background of grape leaf spots in the field is complex, with rich edge textures in small diseased areas and similar disease symptoms, seriously affecting the accuracy of disease segmentation. To address this challenge, common segmentation methods typically include the following:

[0003] (1) Convolutional Neural Network

[0004] Convolutional neural networks (CNNs) are widely used in agriculture, such as the Deeplab series, Unet, and PSPnet. These network architectures are mainly designed to improve the segmentation performance of the model by increasing network depth or introducing residual learning. Although the above networks have achieved great success in extracting lesion features, they have some limitations on their performance: (1) fixed convolutional kernels constrain the size of the receptive field, resulting in poor segmentation of natural images with complex backgrounds; (2) local connectivity causes them to ignore the criticality of long-distance semantic information interaction; and (3) small target disease regions are difficult to segment accurately. To solve the above problems, it is necessary to extract local features while gaining a deeper understanding of global information and to explore more scene-level semantic information in the entire natural scene.

[0005] (2) Transformer

[0006] The Transformer has further advanced the field of vision and demonstrated superior performance compared to convolutional neural network-based models in segmentation tasks. Due to its robust self-attention mechanism, the Transformer can explicitly model global contextual information when given high-resolution natural images with complex backgrounds. Some researchers have improved the performance of plant lesion segmentation by modifying the Transformer. Furthermore, many works have extracted low-resolution features of lesions by introducing convolution operators, downsampling feature maps, employing pyramidal hierarchies, and redesigning labels. However, they still follow a sequential topology, progressively making each stage output features of the same resolution scale, but neglecting the information interaction between different resolutions within the same stage, resulting in the inability to generate high-quality segmented images. When identifying grape leaf lesions with complex backgrounds and small targets, this leads to reduced segmentation performance and loss of edge detail. Empirically, high-resolution feature maps can capture finer-grained information, especially for grape leaf disease edges. Low-resolution feature maps typically contain stronger semantic representation information, especially for small, difficult-to-segment disease areas. For this reason, maintaining both high-resolution feature maps and deep semantic information is crucial for efficiently handling segmentation tasks.

[0007] (3) Attention mechanism

[0008] The goal of attention mechanisms is to concentrate the network's attention on the most important small subset of data by increasing the weight of certain parts of the input data and decreasing the weight of others. Attention mechanisms are broadly divided into two branches: spatial attention and channel attention. Different types of attention serve different functions. For example, spatial attention aims to enhance the spatial feature representation of key regions. Channel attention aims to model the correlations between different channels. However, existing popular Transformer architectures neglect the importance of channel dimensionality adaptability. The challenges in grape leaf lesion image segmentation can be addressed by inheriting the advantages of both channel attention and spatial attention. Summary of the Invention

[0009] The purpose of this invention is to tailor a model for grape leaf lesion image segmentation, called the cross-resolution Transformer model, to solve the problem of grape leaf lesion image segmentation in complex backgrounds in natural field environments. Compared with previous Transformer-based models that only considered acquiring high-resolution features or semantic information, the cross-resolution Transformer model simultaneously considers preserving high-resolution features and strong semantic information.

[0010] Specifically, the cross-resolution Transformer model has an encoder-decoder structure. The encoder consists of four stages, with i parallel connected multi-resolution Transformer blocks distributed in a pyramid shape in the i-th stage. The decoder is a hamburger decoder.

[0011] Following the design philosophy of parallel Transformers, the input is first downsampled using CONV-BN-ReLU blocks to effectively extract low-resolution feature maps.

[0012] Specifically, the network architecture of the multi-resolution Transformer block includes a large kernel mining attention mechanism, a multi-path feedforward network, and a cross-resolution fusion strategy.

[0013] Specifically, the large kernel mining attention mechanism has a convolution kernel size of 11×11 to expand the receptive field. After the input data is embedded and reshaped by overlapping features, it is fed into the large kernel mining attention mechanism, and the output features of the attention mechanism are calculated by the Hadamard operator.

[0014] Specifically, given input data Where H×W and C represent the number of input tags and the channel feature dimension, respectively, after overlapping feature embedding and reshaping, they are fed into the large kernel attention mechanism. The similarity-based score matrix A is calculated through a large kernel depthwise convolution with a kernel size of k×k. In this invention, the convolution kernel size is set to k=11 to effectively expand the receptive field. Based on the self-attention mechanism, the matrix... Multiplying the result by the embedded input yields the numerical value V. In the i-th stage, the specific details are as follows:

[0015] A = DWConv k×k (W i X)

[0016]

[0017] in and The weight matrix is ​​obtained by linear projection, where matrix A represents the similarity or correlation between each pair of input labels, and DWConv(·) represents the depthwise separable convolution operation. The values ​​of V and A, calculated by the Hadamard operator, are used as the attention output, with the specific formula as follows:

[0018] Attention(X) = A ⊙ V

[0019] Compared to attention mechanisms where computational complexity increases quadratically, large-kernel convolution is a fully convolutional mechanism, thus its complexity and parameter count maintain a linear growth. Large-kernel mining attention mechanisms enable effective information interaction between channels. When high-resolution images are used as input, such as grape leaf disease images, the coding space and channel information weight matrix generated by the large-kernel mining attention mechanism can adapt to the input.

[0020] Specifically, the multi-path feedforward network considers the importance of multi-level semantic mining and is implemented by directly applying k×1 and 1×k (k=3,5) bi-branch convolutional pairs and an expansion ratio r (r=4). The multi-path feedforward network can be described as follows:

[0021] x3=Conv 3×1 (Conv 1×3 (Linear(x in )))

[0022] x5 = Conv 5×1 (Conv 1×5 (Linear(x in )))

[0023] x out =Cat(x3,x5)+x in

[0024] LN normalization and the activation layer (GELU) are omitted, x in x represents the features output from the attention mechanism mined from the big kernel. out This represents the features output from the multipath feedforward network. The design of multipath feedforward networks can further capture receptive fields at different scales, which helps to improve the ability to aggregate multi-scale information.

[0025] Experiments show that exchanging information between different resolution scales helps to generate high-quality, high-resolution images. Therefore, this invention adopts a cross-resolution fusion strategy to transfer semantic information between two consecutive stages, thereby achieving information interaction between adjacent stages.

[0026] Specifically, the cross-resolution fusion strategy constructs a static two-dimensional matrix through binary loops and fuses semantic information through upsampling or downsampling. The semantic information of the low-resolution feature map branch is upsampled to the high-resolution feature map branch to extract semantic features with a larger receptive field. The high-resolution feature map is downsampled to the low-resolution feature map to preserve more image details, thereby achieving accurate segmentation of small target lesions with complex backgrounds.

[0027] More specifically, let the feature resolution of the input branch be j and the feature resolution of the output be n. To obtain high-level features with a larger receptive field, the low-resolution features are upsampled and merged into the high-resolution features. That is, when j > n, a 1×1 convolution is used to keep the number of channels in layer j and layer n the same, and the spatial dimension is upsampled by nearest interpolation. To enable the low-resolution features to retain more image details, the low-resolution features are merged with the downsampled high-resolution features. That is, when j < n, a depthwise separable convolution with a stride of 2^(j - n) + 1 is used to reduce the spatial dimension of the high-resolution features and match the output number of channels. When j = n, a skip connection is used to directly output the features. The cross-resolution semantic fusion strategy inherits the advantages of high-resolution representations and stronger semantic information of low-resolution, which is beneficial for accurate segmentation of small diseases with complex backgrounds.

[0028] Specifically, the hamburger decoder uses matrix factorization methods to model global spatial information, aggregates the context information of the last three layers to fuse feature information of different scale resolutions. Since there are more low-level features in the first stage, aggregating the first stage will result in a higher computational cost. Therefore, only the context information of the last three stages is aggregated, and the information is aggregated from low-resolution features and high-resolution features.

[0029] The advantages of the present invention are that, compared with the previous Transformer models that separately consider obtaining high-resolution features or semantic information, the cross-resolution Transformer model simultaneously considers retaining high-resolution features and strong semantic information. A novel cross-resolution Transformer architecture is proposed, which performs cross-resolution information transfer in a parallel manner and utilizes the advantages of Transformer to improve representation learning and extract robust semantic information; a large kernel mining attention mechanism is introduced, where large kernel convolution is used to reshape the pixel weight matrix, adaptively process channel and spatial information without increasing the computational cost, and mine context information from the entire scene; a multi-path feed-forward network and a hamburger decoder are designed to further expand the multi-scale receptive field and improve the ability of multi-scale information aggregation. Applying the present invention can effectively solve the problem of grape leaf disease spot segmentation in the complex background of natural fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is an overall architecture diagram of a cross-resolution Transformer model;

[0031] Figure 2 is a framework diagram of a Transformer block; ​​​​​ Figure 4 This is a schematic diagram of a cross-resolution fusion strategy. Detailed Implementation

[0034] The overall framework diagram of the grape leaf lesion image segmentation method based on the multi-resolution Transformer model is as follows: Figure 1 As shown, this model is an encoder-decoder model; Figure 2 This is a diagram of the Transformer block framework of the present invention; as shown below. Figure 3 The diagram shown is a schematic of the decoder of this invention, namely the hamburger decoder; Figure 4 This is a schematic diagram of the cross-resolution fusion strategy employed by the model. During the training phase, the experiments of this invention and others were deployed on PyTorch and the mmsegmentation library for semantic segmentation experiments. All models were trained on NVIDIA Tesla V100 GPUs. To ensure fairness in comparison, this invention follows the same training strategy as previous work. Specifically, the resolution of the training images was cropped to 1024×1024. During the training phase, AdamW with a weight decay of 0.01 was used to optimize the model of this invention. A "poly" LR strategy was used to train the LRT (lr = baselr × (1-epoch / maxiter)). power ), where the "poly" LR policy factor is set to 1, and the initial learning rate is 6×10. -5 A total of 160,000 iterations were performed.

[0035] This invention was trained and evaluated on three datasets: the Field-PV dataset, the Plant Village dataset, and the Syn-PV dataset. The Field-PV dataset was collected by the Institute of Forestry and Fruit Trees, Beijing Academy of Agricultural and Forestry Sciences, China, using an OLYMPUS OM-D camera, containing 400 raw images of natural scenes containing grape grad mold disease. The Plant Village dataset is a public and unbiased dataset specifically for crop disease and pest identification. It consists of 54,303 high-resolution images, including images of different disease categories and healthy leaves from 38 plant species. These images were obtained in a controlled laboratory. This invention used 1,383 images of grape black measles and 1,180 images of grape black rot from this dataset for training, and manually labeled them. The Syn-PV dataset is a natural field image synthesized from Plant Village segmented images obtained from the controlled laboratory through background replacement. The background replacement method was used to synthesize grape disease images with complex backgrounds.

[0036] All datasets were generated by manually labeling diseased and leaf areas in collected images using the labelme tool. The labeled data was saved in JSON format and then converted to the PASCALVOC 2012 data format with semantic labels for foreground and background objects. This invention utilizes the Augmentor module for geometric transformations, such as random left / right flipping, random cropping, random sampling, and color and brightness enhancement or reduction. During training, this invention applies data augmentation methods provided by the semantic segmentation library mmsegmentation.

[0037] To evaluate the effectiveness of the cross-resolution Transformer (CRFormer) model, this invention compares it with other image segmentation methods. Four metrics—precision, IoU, Recall, and Dice—are used to measure model performance, with bold indicating the maximum value and underlined indicating the suboptimal result. Furthermore, the parameters and gigabit floating-point operations per second (FLOPs) of each model are analyzed, and the results are detailed in Tables 1-5.

[0038] Table 1. Quantitative comparison results of CRFormer and other segmentation methods on the Plant Village dataset for black measles and black rot segmentation.

[0039]

[0040] Table 2 shows the quantitative comparison results of CRFormer and other segmentation methods for background and grape leaf segmentation on the Plant Village dataset.

[0041]

[0042] Table 3 shows the quantitative comparison results of CRFormer and other methods for black measles and black rot segmentation on the Syn-PV dataset.

[0043]

[0044] Table 4 shows the quantitative comparison results of CRFormer and other segmentation methods for background and grape leaf segmentation on the Syn-PV dataset.

[0045]

[0046] Table 5 shows the quantitative comparison results of CRFormer and other segmentation methods on the Field-PV dataset for background and gray mold segmentation.

[0047]

[0048] Experimental results show that CRFormer outperforms state-of-the-art Transformer and deep learning-based methods in grape leaf lesion segmentation. Considering image segmentation performance and training and operating costs, this invention achieves optimal performance in complex grape leaf lesion segmentation tasks.

Claims

1. A method for segmenting grape leaf lesion images based on a cross-resolution Transformer model, characterized in that, The cross-resolution Transformer model has an encoder-decoder structure. The encoder consists of four stages, with i parallel-connected multi-resolution Transformer blocks arranged in a pyramid shape in the i-th stage. The decoder is a hamburger decoder. The network architecture of the multi-resolution Transformer block includes a large kernel mining attention mechanism, a multi-path feedforward network, and a cross-resolution fusion strategy; The kernel size of the large kernel mining attention mechanism is set to 11×11 to expand the receptive field. After the input data is embedded and reshaped by overlapping features, it is fed into the large kernel mining attention mechanism, and the output features of the attention mechanism are calculated by the Hadamard operator. The multi-path feedforward network considers the importance of multi-level semantic mining and is implemented using k×1 and 1×k bi-branch convolutional pairs and a spreading ratio r, where k is set to 3 and 5, and r is set to 4. The specific formula is as follows: x3=Conv 3×1 (Conv 1×3 (Linear(x in ))) x5=Conv 5×1 (Conv 1×5 (Linear(x in ))) x out =Cat(x3,x5)+x in LN normalization and GELU activation layer are omitted, x in x represents the features output from the attention mechanism mined from the big kernel. out This represents the features output from the multipath feedforward network; The cross-resolution fusion strategy constructs a static two-dimensional matrix through binary loops and fuses semantic information through upsampling or downsampling. The semantic information of the low-resolution feature map branch is upsampled to the high-resolution feature map branch to extract semantic features with a larger receptive field. The high-resolution feature map is downsampled to the low-resolution feature map to preserve more image details, thereby achieving accurate segmentation of small target lesions with complex backgrounds. The hamburger decoder uses matrix factorization to model global spatial information and aggregates the context information of the last three layers to fuse feature information at different scale resolutions.

Images