A u-shaped perception lightweight transformer method for segmenting grapevine leaf small lesions

Abstract

The present application belongs to the field of agricultural information technology, and particularly relates to a U-shaped perception lightweight Transformer method for segmenting grape leaf small disease spots. A lightweight convolutional neural network MobileNetV2 is adopted to extract multi-scale feature information through U-shaped pyramid down-sampling; a context perception enhancement module is used to extract low-frequency global features and high-frequency local feature maps; a token aggregation strategy is introduced to reduce the loss of detailed information caused by direct aggregation of low-frequency and high-frequency features; and the aggregated tokens are directly transmitted to a lightweight segmentation head to implement a segmentation task. The present application can achieve balanced efficiency and speed to solve the problem of grape leaf small disease spot segmentation in a natural field complex background.

Description

Technical Field

[0001] This invention belongs to the field of agricultural information technology, and specifically relates to a U-shaped sensing lightweight Transformer method for segmenting small lesions on grape leaves. Background Technology

[0002] Grape leaf spots are a major factor contributing to reduced grape yields and quality. Furthermore, these spots can rapidly spread fungi throughout the vineyard, triggering epidemics across the entire field. Lightweight segmentation models, by assigning labels to each pixel to quickly understand disease distribution, facilitate rapid diagnosis and monitoring of disease trends on leaves, improving treatment efficiency and reducing costs through targeted management measures. However, heavyweight models are not ideal for these rapidly spreading plant diseases requiring timely segmentation and are difficult to deploy on resource-constrained hardware. To improve segmentation efficiency, segmentation models need to be designed to be accurate, lightweight, and fast.

[0003] (1) Convolutional Neural Network

[0004] Lightweight vision tasks have been overwhelmingly dominated by Convolutional Neural Networks (CNNs). Their inherent inductive bias and weight-sharing properties allow models to learn representations with fewer parameters. However, they suffer from several limitations: 1. The local connectivity of CNNs often prevents the modeling of long-term dependencies, thus ignoring fine-grained semantic information in complex backgrounds. 2. Fixed convolutional kernels and weights lead to a loss of detail, making it impossible to extract pixel-level semantics of small target lesions. Benefiting from self-attention mechanisms, Transformer-based methods have demonstrated excellent long-range modeling capabilities, offering an alternative to CNNs for learning global visual representations. However, the heavy weights and time-consuming computational mechanisms are not ideal for inference in real-world industrial deployments.

[0005] (2) Transformer

[0006] To reduce computational efficiency, many Transformer variants strive to liberate models from the burden of time-consuming computations. Furthermore, many works extract low-resolution features of lesions by introducing convolutional operators, downsampling feature maps, employing pyramidal hierarchies, and redesigning labels. When the model is compressed to a mobile-friendly size, the segmentation performance is sacrificed, which seems like a disastrous failure for lightweight models. CNNs have demonstrated impressive performance in their inherently biased architectural design; therefore, recent work attempts to embed the advantages of CNNs into Transformers to achieve a superior accuracy-efficiency tradeoff. Among them, MobileFormer leverages MobileNet and Transformer to achieve bidirectional fusion of local and global information at a lower computational cost. Unfortunately, the aforementioned methods focus on capturing low-frequency global information, neglecting the importance of high-frequency local information, which is helpful in extracting features from small lesions.

[0007] (3) Feature aggregation strategy

[0008] The weakness of directly aggregating high-resolution and low-frequency feature information is that details are easily obscured or lost. Some pioneering works have explored various aggregation schemes to overcome this problem. ASPP utilizes multiple parallel branches with different dilation rates for dilated convolutions, enabling the model to aggregate local and global contextual information without a significant increase in computational complexity. PPM in PSPNet captures multi-scale contextual information by performing pyramid pooling on the input feature maps, enhancing the model's perception of features at different scales and producing finer segmentation results than ASPP. To obtain even finer contextual information, subsequent variants, such as DAPPM, have been derived for application, integrating features through larger convolutional kernels and deeper information flows. However, the depth information in the above methods is not processed in parallel, and the number of channels per scale is relatively large, meaning the computational cost is relatively high. Summary of the Invention

[0009] This invention discloses a U-shaped perception lightweight Transformer method for segmenting small lesions on grape leaves. It employs a lightweight convolutional neural network, MobileNetV2, to extract multi-scale feature information through U-shaped pyramid downsampling; extracts low-frequency global features and high-frequency local feature maps through a context-aware enhancement module; introduces a token aggregation strategy to reduce the loss of detail information caused by direct aggregation of low-frequency and high-frequency features; and directly transmits the aggregated tokens to the lightweight segmentation head to perform the segmentation task.

[0010] Preferably, the U-shaped pyramid uses MobileNetV2 to extract feature information and uses the average pooling operator to reduce the resolution of the original tokens, and concatenates the tokens of different scales along the channel dimension to generate new tokens. These new tokens are used as input to the context-aware enhancement module. Since the multi-scale tokens are downsampled to a smaller resolution, i.e. the new tokens have a large number of channels, the extraction of multi-scale feature information has low computational complexity.

[0011] Preferably, the context-aware enhancement module includes a prototype-aware branch and a pixel-aware broadcast branch.

[0012] Preferably, the prototype-aware branch downsamples K and V to reduce matrix operations; convolutional layers exchange information between tokens along the spatial dimension to reduce the number of reshaping operations; nonlinear activation layers are replaced by ReLU6 and GELU; batch normalization is added to each convolution, which is faster than the layer normalization method used in inference; fine-grained semantic information in the tokens is contained in the residual mapping of the Transformer. The prototype-aware branch effectively obtains the global receptive field and enhances low-frequency representations with lower computational cost.

[0013] Preferably, the pixel-aware broadcast branch employs a convolutional attention mechanism to effectively mine context weights by sharing weights and pixel-aware weights; a linear layer is used to generate the key K, query Q, and value V, as follows:

[0014] Q,K,V=Linear(X in )

[0015] Where X in This represents the features input from the U-shaped pyramid.

[0016] Preferably, the pixel-aware broadcast branch comprises the following steps:

[0017] Step 1: Extract local features using the depthwise convolution (DWconv) operator and apply shared weights to V, as shown in the following formula:

[0018] V = DWconv(V)

[0019] Step 2: Perform local enhancement processing on Q and K with pixel-aware weights; use two translation-invariant convolutions to obtain local information for Q and V respectively; calculate the values ​​of Q and K using the Hadamard product and use them as output; replace the Softmax in traditional attention with Tanh and Awish to obtain pixel-aware weights between -1 and 1; adopt a gating mechanism to obtain pixel-aware weights, which has stronger non-linearity. Stronger non-linearity means higher quality pixel-aware weights, as described below:

[0020] Q l =DWconv(Q)

[0021] K l =DWconv(K)

[0022] Atten l =Linear(Swinsh(Linear(Q) l ⊙K l )))

[0023]

[0024] The generated, stronger non-linear weights are aggregated with other pixels, and local features are enhanced through a Hadamard product operation. The output map is defined as:

[0025] X local =Attn⊙V

[0026] Preferably, the token aggregation strategy effectively aggregates low-frequency global information and high-frequency local information, reducing the number of channels at each scale; global average pooling with different convolution kernels and strides is used to obtain feature maps at different image resolutions. Channel dimensions at different scales are transformed using 1×1 convolutions, and the feature maps are upsampled. Then, 3×3 convolutions are used to aggregate the original features with background information at different scales. Finally, 1×1 convolutions are used to concatenate and compress the feature maps. Furthermore, a 1×1 residual mapping is introduced for optimization. Assuming x is the input, the features at each scale can be represented as:

[0027]

[0028] Here, Up represents upsampling.

[0029] The advantages of this invention lie in its proposed lightweight U-shaped perceptual Transformer, which downsamples tokens of different scales to a smaller scale as input, while inheriting the advantages of CNN and Transformer. The core component, the perceptual enhancement module, employs a parallel architecture in small-scale tokens (n) to achieve superior cost-effectiveness. The perceptual enhancement module consists of two branches: a prototype perceptual branch learns low-frequency global information through downsampling K and V, while the pixel-perceptual broadcast branch uses a gating mechanism to enhance nonlinearity and mines high-frequency local information through shared weights and context-aware weights. A token aggregation strategy is designed to compensate for sacrificed detail information without increasing the number of parameters. Applying this invention can achieve a balance between efficiency and speed to solve the problem of segmenting small lesions on grape leaves in complex natural field backgrounds. Attached Figure Description

[0030] Figure 1This is an overall architecture diagram of a U-shaped sensing lightweight Transformer method for segmenting small lesions on grape leaves;

[0031] Figure 2 This is a schematic diagram of pixel-aware broadcast branch;

[0032] Figure 3 This is a diagram illustrating the token aggregation strategy. Detailed Implementation

[0033] The overall framework diagram of a U-shaped sensing lightweight Transformer method for segmenting small lesions on grape leaves is shown below. Figure 1 As shown; Figure 2 This is a schematic diagram of the pixel-aware broadcast branch in the context-aware enhancement module of the present invention; as shown. Figure 3 The diagram shown is a schematic of the token aggregation strategy disclosed in this invention.

[0034] During the training phase, the experiments of this invention and other comparative methods were all deployed on PyTorch and the mmsegmentation library for semantic segmentation experiments. All models were trained on NVIDIA Tesla V100 GPUs. To ensure fairness in the comparison, this invention followed the same training strategy as previous work. Specifically, images were randomly cropped to 512×512. During the training phase, AdamW with a weight decay of 0.01 was used to optimize the model of this invention. A "poly" LR strategy was used to train the LRT.

[0035] (lr=baselr×(1-spoch / maxiter) power ), where the "poly" LR policy factor is set to 1, and the initial learning rate is 6×10. -6 A total of 160,000 iterations were performed.

[0036] This invention evaluated the designed architecture on three datasets: the Field-PV dataset, the Plant Village dataset, and the Syn-PV dataset. The Field-PV dataset was acquired using an OLYMPUSOM-D camera at the Institute of Forestry and Fruit Trees, Beijing Academy of Agricultural and Forestry Sciences, China. A total of 400 raw images containing natural scenes of grape gray mold were captured. Plant Village is an open and impartial dataset specifically for crop disease and pest identification. This dataset consists of 54,303 high-resolution images, including different disease categories and healthy leaves of 38 plant species. These images were obtained in a controlled laboratory. We utilized 1,383 images of grape black measles and 1,180 images of grape black rot. Syn-PV is a synthesis of natural field images from Plant Village segmented images obtained from a controlled laboratory through background replacement. Background replacement was used to synthesize grape disease images with complex backgrounds. All datasets were manually annotated with diseased areas and leaf regions using the labelme tool. Annotated data is saved in JavaScript object (.json) format. The data is then converted to the PASCAL VOC 2012 format, which includes semantic labels for foreground and background objects. This invention utilizes the Augmentor module for geometric transformations such as random left / right flipping, random cropping, random sampling, and color and brightness enhancement or reduction. During training, this invention applies basic yet powerful data augmentation methods from the semantic segmentation library mmsegmentation.

[0037] To evaluate the effectiveness of the U-shaped perceptual Transformer, the model was compared with other segmentation methods. These included three classic segmentation methods: DeeplabV3+, UNet, and PSPNet; four heavyweight Transformer-based segmentation methods: PVT2, Dual-ViT, Segformer, and Segnext; and seven lightweight Transformer-based segmentation methods: Seaformer, AFFormer, Poolformer, EfficientFormer, LVT, NextViT, and Topformer.

[0038] The evaluation metrics used are accuracy, IoU, recall, and Dice to measure model performance. Additionally, the parameters, gigaflops (GFLOPs), Fps, and memory usage of each model were analyzed.

[0039] Table 1. Quantitative comparison of grape leaf diseases on the Plant Village dataset based on CNN and Transformer methods.

[0040]

[0041] Table 2. Quantitative comparison of grape leaves and background on the Shuzun dataset based on CNN and Transformer methods.

[0042]

[0043] Table 3. Quantitative comparison of grape leaf diseases and background based on CNN and Transformer methods on the field-PV dataset.

[0044]

[0045] Experimental results show that the segmentation performance of this invention outperforms state-of-the-art Transformer methods and deep learning-based methods. Considering both image segmentation performance and training and operating costs, this invention achieves optimal performance and a balance between segmentation performance and speed in the complex task of segmenting small lesions on grape leaves.

Claims

1. A lightweight Transformer method for segmenting small lesions on grape leaves using a U-shaped sensing mechanism, characterized in that... The lightweight convolutional neural network MobileNetV2 is used to extract multi-scale feature information through U-shaped pyramid downsampling; low-frequency global features and high-frequency local feature maps are extracted through a context-aware enhancement module; a token aggregation strategy is introduced to reduce the loss of detail information caused by direct aggregation of low-frequency and high-frequency features; the aggregated token is directly transmitted to the lightweight segmentation head to perform the segmentation task. The U-shaped pyramid uses MobileNetV2 to extract feature information and uses the average pooling operator to reduce the resolution of the original tokens. It then concatenates the tokens of different scales along the channel dimension to generate new tokens. These new tokens are used as input to the context-aware enhancement module. The context-aware enhancement module includes a prototype-aware branch and a pixel-aware broadcast branch; The prototype-aware branch downsamples K and V, the convolutional layers exchange information between tokens along the spatial dimension, the nonlinear activation layers are replaced by RELU6 and GELU, batch normalization is added to each convolution, and the fine-grained semantic information in the tokens is contained in the residual mapping of the Transformer. The pixel-aware broadcast branch employs a convolutional attention mechanism to effectively mine context weights by sharing weights and pixel-aware weights; a linear layer is used to generate the key K, query Q, and value V, as shown in the following equation: Q,K,V=Linear(X in ) Where X in This represents the features input from the U-shaped pyramid; The pixel-aware broadcast branch consists of the following steps: Step 6.1: Extract local features using the depthwise convolution (DWconv) operator and apply shared weights to V, as shown in the following formula: V = DWconv(V) Step 6.2: Perform local enhancement processing on Q and K with pixel-aware weights; use two translation-invariant convolutions to obtain local information of Q and V respectively; calculate the values ​​of Q and V through the Hadamard product and use them as outputs; replace the Softmax in the traditional attention with Tanh and Awish; and use a gating mechanism to obtain pixel-aware weights.

2. The method according to claim 1, characterized in that, The token aggregation strategy effectively aggregates low-frequency global information and high-frequency local information, reducing the number of channels at each scale; it uses global average pooling with different convolution kernels and strides to obtain feature maps for different image resolutions.

Images