An automatic weed segmentation method based on DA-Net
Through the branch and spatial attention module of the DA-Net deep learning model, combined with the ResNet101 network, the problem of difficulty in automatic weed distinction in the existing technology is solved, high-precision weed segmentation is achieved, reducing the use of herbicides, and improving the Split effect.
Patent Information
- Application Number
- CN202111641200.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-12-29
AI Technical Summary
The prior art is difficult to effectively and automatically distinguish weeds from crops, resulting in large-scale use of herbicides and causing environmental and crop pollution.
Using a deep learning model based on DA-Net, we automatically segment weeds by building branch attention modules and spatial attention modules, combined with ResNet101 network, multi-level and long-distance dependencies between high-level features and low-level features.
High-precision weed segmentation is achieved, reducing the use of herbicides and improving segmentation effect, especially Mean IoU performance on beet and carrot weed datasets.
Smart Images

Figure CN114299091B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of precise segmentation of agriculture, and in particular relates to an automatic weed segmentation method based on DA-Net. Background Art
[0002] Weeds are an important cause of crop yield loss. Traditional methods usually use large-scale herbicides to control weeds. The weed segmentation task is to use image information to automatically distinguish weeds from crops at the pixel level. Applying this technology to unmanned robots can reduce environmental pollution and crop contamination. In this paper, we use deep learning technology in precision agriculture to accurately distinguish weeds from crops, which can greatly reduce the use of herbicides. Summary of the invention
[0003] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention proposes a method for automatic weed segmentation based on DA-Net. Automatic weed segmentation is achieved, which greatly reduces the use of herbicides. To this end, the present invention proposes a deep learning model of DA-Net, and the specific steps are as follows:
[0004] Step 1: Preprocessing of experimental data;
[0005] Step 2: Build the DA-Net network model;
[0006] Step 3: Train the DA-Net network model;
[0007] Step 4: Use the trained DA-Net network model to segment the weeds in the land and evaluate the segmentation effect.
[0008] The step 1 specifically comprises the following steps:
[0009] Step 1.1: In order to improve the generalization ability of the model, the image is first normalized;
[0010] Step 1.2: Crop the normalized image to a size of 352×480;
[0011] Step 1.3: Perform data enhancement on the cropped image and expand the training samples to reduce overfitting. Specifically, this is done by randomly applying geometric transformation methods such as affine transformation, translation, horizontal flipping, and scaling; or applying pixel transformation methods such as motion blur, median filtering, kernel blur, and random bright contrast.
[0012] Step 1.4 divides the training set, validation set, and test set into a ratio of 8:1:1.
[0013] The step 2 comprises the following steps:
[0014] Step 2.1 uses ResNet101 as the backbone network in the encoding stage to extract semantic information;
[0015] Step 2.2 builds a branch attention module and adds the branch attention module to the backbone network in the encoding phase.
[0016] Res-2 of the second layer of the backbone network and high-level features Res-3, Res-4 and Res-5 pass through branch attention module I together; Res-3 of the third layer and high-level features Res-4 and Res-5 pass through branch attention module II together; Res-4 of the fourth layer will pass through branch attention module III together with high-level features Res-5.
[0017] Before passing through the branch attention module, bilinear interpolation is used to restore the high-level features to the image resolution.
[0018] Furthermore, high-level features are concatenated with low-level features and passed through the global average pooling layer G gp Get a one-dimensional channel feature vector.
[0019] Furthermore, the one-dimensional channel feature vector is obtained by F cr (consisting of a 1×1 convolution layer, a RELU activation function, and a 1×1 convolution layer), and finally sent to the sigmoid activation function to obtain the branch attention map.
[0020] Furthermore, the branch attention map is multiplied with the high-level features after the resolution is restored to obtain new high-level features. The new high-level features are multiplied with the low-level features to obtain the output features.
[0021] The low-level feature Res-3 of the third layer Res-3 in the encoding stage is recorded as The high-level features Res-4 and Res-5 of the fourth layer Res-4 and the fifth layer Res-5 are respectively denoted as Where H and W are the length and width of the image, and C is the number of channels;
[0022] The high-level features L4 and L5 are used to restore the image resolution using bilinear interpolation, and L4 and L5 are converted to
[0023] Further, the low-level feature L3 and the restored high-level features L4 and L5 are concatenated to obtain The concatenated feature U is sent to the global pool to obtain The calculation formula is as follows:
[0024]
[0025] Among them, G gp is the global average pooling layer; U i,jRepresents the value with coordinates (i, j);
[0026] Furthermore, Z is sequentially sent to a 1×1 convolutional layer, a RELU activation function, and a 1×1 convolutional layer, and finally sent to a sigmoid activation function to obtain a branch attention graph S. The calculation formula is as follows:
[0027] S=σ(F cr (Z)) (2)
[0028] where S = [s1, s2, ..., s 12C ], σ is the sigmoid activation function. F cr It includes a 1×1 convolution layer, a RELU activation function, and a 1×1 convolution layer;
[0029] Multiply the branch attention map S with the high-level features L4 and L5 after the resolution is restored to obtain the new features and and Splicing New Features and The calculation formula is as follows:
[0030]
[0031]
[0032] Where S1=[s1,s2,...,s 4C ] is the branch attention map of the high-level feature L4 after restoring the resolution; S2 = [s 4C+1 ,s 4C+2 , ..., s 12C ] is the branch attention map of the high-level feature L5 after restoring the resolution; F scale (L4, S1) refers to the channel multiplication between the weight S1 and the high-level feature L4 after the resolution is restored;
[0033] Furthermore, Multiply with the low-level feature L3 to get the final output feature The calculation formula is as follows:
[0034]
[0035] Among them, ⊙ represents the multiplication of matrix elements;
[0036] Step 2.3: The final feature Y after fusion by the branch attention module is sent to the global convolutional network (GCN) for feature extraction;
[0037] Furthermore, the global convolutional network is composed of two convolutional branches: 1× K+K×1 convolution and K×1+1×K convolution.
[0038] Step 2.4: Build a spatial attention module;
[0039] The features of the second layer and the high-level features of the third layer after deconvolution pass through the spatial attention module together; the features of the third layer and the high-level features of the fourth layer after deconvolution pass through the spatial attention module together; the features of the fourth layer and the high-level features of the fifth layer after deconvolution pass through the spatial attention module together; and only the features of the fifth layer themselves pass through the spatial attention module.
[0040] Before passing through the spatial attention module, the high-level features of the decoding stage are concatenated with the low-level features to obtain Where H and W are the length and width of the image, and C is the number of channels;
[0041] Further, the characteristics After three 1×1 convolutional layers, we get
[0042] Furthermore, Q, K and V are transformed into Where N = H × W. Multiply the transposed Q by K, and pass the obtained features through a layer of Softmax function to obtain the spatial attention map. The calculation formula is as follows:
[0043]
[0044] in,
[0045] Furthermore, multiply V by the transposed T to obtain Transform E into And send it to the 1×1 convolution layer. The final result Y is calculated as follows:
[0046] Y=αE+X (8)
[0047] Among them, α is a learnable variable;
[0048] Step 2.5: Use deconvolution to restore image resolution during the decoding stage;
[0049] The step 3 comprises the following steps:
[0050] Step 3.1: Use weighted cross entropy loss function during training;
[0051] Step 3.2: Use the stochastic gradient descent optimization algorithm SGD during training;
[0052] Step 3.3: The ResNet101 network weights in the DA-Net encoding phase use the weights pre-trained on the ImageNet dataset;
[0053] Step 3.4: Use the poly learning rate strategy during training, and the learning rate update formula is:
[0054]
[0055] Among them, learnrate is the learning rate of the current epoch, initial_rate is the initial learning rate, iter is the current epoch number, and total_iter is the total epoch number.
[0056] The step 4 specifically comprises the following steps:
[0057] Step 4.1: Introduce the IoU coefficient and ACC coefficient to evaluate the segmentation effect. The definitions of the two are as follows:
[0058]
[0059]
[0060] Among them, TP represents true positive; TN represents true negative; FP represents false positive; and FN represents false negative.
[0061] Compared with the prior art, the beneficial results of the present invention are:
[0062] The present invention realizes automatic weed segmentation by establishing a deep learning model of DA-Net, reducing the large-scale use of herbicides. Compared with other U-shaped network structures, this method can better promote the fusion of high-level features and low-level features, and help the two establish multi-level and long-distance dependencies. Thus, high-precision weed segmentation can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 This is the network structure diagram of DA-Net.
[0064] Figure 2 This is the structural diagram of the branch attention module.
[0065] Figure 3 This is the structural diagram of spatial attention.
[0066] Figure 4 An example of DA-Net segmenting beet weeds. DETAILED DESCRIPTION
[0067] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.
[0068] In order to solve the problems encountered in agricultural weeding, the present invention proposes an automatic weed segmentation method based on DA-Net. DA-Net is a U-shaped architecture based on a dual attention mechanism, which consists of an encoder, a branch attention module, a decoder and a spatial attention module. First, a one-dimensional image is sent to the encoder to extract semantic information. The four layers of features extracted by ResNet101 pass through the branch attention module respectively, and each layer will be fused with its higher layer. The fused features are sent to the global convolutional network (GCN) for feature extraction; in the decoder part, each layer of features will pass through the spatial attention module with the next layer of features; finally, deconvolution is used to restore the image resolution. Since high-level features have high-level semantic information and low resolution, and low-level features have low-level semantic information and high resolution; ordinary deep learning models only fuse the two equally, and the branch attention module allows the network to select the appropriate branch for fusion. In the decoding stage, many traditional deep learning models only use linear aggregation to help high-level features and low-level features establish multi-level and long-distance dependencies, but the linear aggregation method may not be enough to provide neurons with strong adaptability. Our spatial attention module can effectively help high-level features establish long-distance and multi-level dependencies with low-level features.
[0069] Example 1 Preprocessing of experimental data.
[0070] (1) Normalize the data.
[0071] (2) Crop the image to a size of 352×480.
[0072] (3) Randomly apply geometric transformation methods such as radial transformation, translation, horizontal flipping and scaling to the image, and apply pixel transformation methods such as motion blur, median filtering, kernel blur and random bright contrast. Perform data enhancement and expand training samples to reduce overfitting.
[0073] (4) The dataset is divided into training set, validation set and test set in a ratio of 8:1:1.
[0074] Example 2 constructs a DA-Net network model.
[0075] (1) ResNet101 is used as the backbone network in the encoding stage to extract semantic information.
[0076] (2) Construct a branch attention module. The second layer Res-2 will pass through the branch attention module together with the high-level features Res-3, Res-4 and Res-5; the third layer Res-3 will pass through the branch attention module together with the high-level features Res-4 and Res-5; the fourth layer Res-4 will pass through the branch attention module together with the high-level feature Res-5.
[0077] (3) The fused features are sent to the global convolutional network (GCN) for feature extraction.
[0078] To construct the spatial attention module, the features of the second layer will pass through the spatial attention module together with the high-level features of the third layer after deconvolution; the features of the third layer will pass through the spatial attention module together with the high-level features of the fourth layer after deconvolution; the features of the fourth layer will pass through the spatial attention module together with the high-level features of the fifth layer after deconvolution; and the features of the fifth layer will only pass through the spatial attention module itself.
[0079] (4) Deconvolution is used to restore image resolution during the decoding stage.
[0080] Example 3: Training the DA-Net network model.
[0081] (1) SGD optimization is used as the optimization method, and weighted cross entropy is used as the loss function for training. The poly learning rate strategy is adopted, and the learning rate formula is:
[0082]
[0083] Among them, learnrate is the learning rate of the current epoch, initial_rate is the initial learning rate, iter is the current epoch number, and total_iter is the total epoch number.
[0084] (2) The ResNet101 network weights in the DA-Net encoding stage use the weights pre-trained on the ImageNet dataset.
[0085] Example 4 uses the trained DA-Net network model to segment weeds and evaluates the segmentation effect. We conducted experiments on three public beet weed datasets, carrot weed dataset and rice field weed dataset.
[0086] (1) Figure 4 This is an example of DA-Net segmenting weeds. The IoU coefficient and ACC coefficient are introduced to evaluate the segmentation effect. The definitions of the two coefficients are as follows:
[0087]
[0088]
[0089] Among them, TP represents true positive; TN represents true negative; FP represents false positive; and FN represents false negative.
[0090] (2) Table 1 below compares the performance of DA-Net with the current mainstream semantic segmentation framework on the beet weed dataset. Compared with ResNet101-Deeplabv3+, our model has a 1.65% improvement in Mean IoU. For details, see Table 1 for a comparison of the performance of DA-Net and other methods.
[0091] Table 1. Comparison of DA-Net performance with other methods on the beet weed dataset
[0092]
[0093] (3) We conducted a series of ablation experiments on DA-Net on the beet weed dataset to verify the role of the branch attention module and the spatial attention module. Compared with the baseline DA-Net, the average IoU of DA-Net with the branch attention module is 88.76%, an increase of 1.56%. When the branch attention module and the position attention module are integrated together, the average IoU of DA-Net reaches 90.07%, an increase of 2.87% compared with the baseline. For details, please refer to the ablation experiment of DA-Net on the dataset in Table 2.
[0094] Table 2. Ablation experiment of DA-Net on the beet dataset
[0095]
[0096] (4) Table 3 below compares the performance of DA-Net with the current mainstream semantic segmentation framework on the carrot weed dataset. Compared with ResNet101-Deeplabv3+, our model has a 2.43% improvement in Mean IoU. For details, see Table 1 for a comparison of the performance of DA-Net and other methods.
[0097] Table 3. Comparison of DA-Net with other methods on carrot weed dataset
[0098]
Claims
1. A DA-Net-based automatic weed segmentation method, characterized in that The specific steps are as follows: Step 1: Preprocessing of experimental data; Step 2: Build the DA-Net network model; Step 2.1 uses ResNet101 as the backbone network in the encoding stage to extract semantic information; Step 2.2 builds a branch attention module and adds the branch attention module to the backbone network in the encoding phase; Step 2.3 The final feature Y after fusion by the branch attention module is sent to the global convolutional network (GCN) for feature extraction; Step 2.4 builds the spatial attention module; Step 2.5 uses deconvolution to restore the image resolution during the decoding stage; Step 3: Train the DA-Net network model; Step 4: Use the trained DA-Net network model to segment the weeds in the land and evaluate the segmentation effect; Step 2.2 builds a branch attention module and adds the branch attention module to the backbone network in the encoding phase. The specific implementation is as follows: The Res-2 of the second layer of the backbone network and the high-level features Res-3, Res-4 and Res-5 pass through the branch attention module I together; the Res-3 of the third layer and the high-level features Res-4 and Res-5 pass through the branch attention module II together; the Res-4 of the fourth layer and the high-level features Res-5 pass through the branch attention module III together; Before passing through the branch attention module, high-level features are restored to image resolution using bilinear interpolation; The high-level features are concatenated with the low-level features and passed through the global average pooling layer G gp Get a one-dimensional channel feature vector; one-dimensional The channel feature vector passes through F cr , and finally sent to the sigmoid activation function to obtain the branch attention map; F cr It consists of a 1×1 convolution layer, a RELU activation function, and a 1×1 convolution layer; the branch attention map is multiplied by the high-level features after the resolution is restored to obtain new high-level features; the new high-level features are multiplied by the low-level features to obtain the output features.
2. The automatic weed segmentation method based on DA-Net according to claim 1 is characterized in that The low-level feature Res-3 of the third layer Res-3 in the encoding stage is recorded as The high-level features Res-4 and Res-5 of the fourth layer Res-4 and the fifth layer Res-5 are respectively denoted as Where H and W are the length and width of the image, and C is the number of channels; The high-level features L4 and L5 are used to restore the image resolution using bilinear interpolation, and L4 and L5 are converted to 3. The automatic weed segmentation method based on DA-Net according to claim 2 is characterized in that The low-level feature L3 and the restored high-level features L4 and L5 are concatenated to obtain The concatenated feature U is sent to the global pool to obtain The calculation formula is as follows: Among them, G gp is the global average pooling layer; U i,j Represents the value at coordinates (i, j).
4. The automatic weed segmentation method based on DA-Net according to claim 3 is characterized in that Send Z to the 1×1 convolution layer, RELU activation function and 1×1 convolution layer in sequence, and finally send it to the sigmoid activation function to obtain the branch attention map S. The calculation formula is as follows: S=σ(F cr (Z)) (2) where S = [s1, s2, ..., s 12C ], σ is the sigmoid activation function.
5. The automatic weed segmentation method based on DA-Net according to claim 4 is characterized in that Multiply the branch attention map S with the high-level features L4 and L5 after the resolution is restored to obtain the new features and and Splicing New Features and The calculation formula is as follows: Where S1=[s1,s2,...,s 4C ] is the branch attention map of the high-level feature L4 after restoring the resolution; S2 = [s 4C+1 ,s 4C+2 ,...,s 12C ] is the branch attention map of the high-level feature L5 after restoring the resolution; F scale (L4, S1) refers to the channel multiplication between the weight S1 and the high-level feature L4 after the resolution is restored, F scale (L5, S2) refers to the channel multiplication between the weight S2 and the high-level feature L5 after the resolution is restored.
6. The automatic weed segmentation method based on DA-Net according to claim 5 is characterized in that Multiply with the low-level feature L3 to get the final output feature The calculation formula is as follows: Here, ⊙ represents the multiplication of matrix elements.
7. The automatic weed segmentation method based on DA-Net according to claim 1 or 6, characterized in that Step 2.4 builds the spatial attention module, which is implemented as follows: The features of the second layer and the high-level features of the third layer after deconvolution pass through the spatial attention module together; the features of the third layer and the high-level features of the fourth layer after deconvolution pass through the spatial attention module together; the features of the fourth layer and the high-level features of the fifth layer after deconvolution pass through the spatial attention module together; and the features of the fifth layer only pass through the spatial attention module themselves; before passing through the spatial attention module, the high-level features of the decoding stage are spliced with the low-level features to obtain Where H and W are the length and width of the image, and C is the number of channels; feature After three 1×1 convolutional layers, we get Transform Q, K and V into Where N = H × W; multiply the transposed Q by K, and pass the obtained features through a layer of Softmax function to obtain the spatial attention map. The calculation formula is as follows: in, Multiply V by the transposed T and we get Transform E into And send it to the 1×1 convolution layer; the final result Y is calculated as follows: Y=αE+X (8) Where α is a learnable variable.
8. The automatic weed segmentation method based on DA-Net according to claim 1 or 7, characterized in that The step 3 comprises the following steps: Step 3.1: Use weighted cross entropy loss function during training; Step 3.2: Use the stochastic gradient descent optimization algorithm SGD during training; Step 3.3: The ResNet101 network weights in the DA-Net encoding phase use the weights pre-trained on the ImageNet dataset; Step 3.4: Use the poly learning rate strategy during training, and the learning rate update formula is: Among them, learnrate is the learning rate of the current epoch, initial_rate is the initial learning rate, iter is the current epoch number, and total_iter is the total epoch number.
9. The automatic weed segmentation method based on DA-Net according to claim 1 or 8, characterized in that The step 4 is specifically The following steps are involved: The IoU coefficient and ACC coefficient are introduced to evaluate the segmentation effect. The definitions of the two are as follows: Among them, TP represents true positive; TN represents true negative; FP represents false positive; and FN represents false negative.
