A Method for Extracting Agricultural Greenhouses Based on Attention Mechanisms and Lightweight Fully Convolutional Networks

Abstract

This invention relates to a method for extracting agricultural greenhouses based on an attention mechanism and a lightweight fully convolutional network. It solves the problems of slow extraction speed and low efficiency in existing technologies. The method includes: S1, acquiring image data of the target region and performing data preprocessing; S2, creating binary greenhouse samples for the target region; S3, constructing a greenhouse recognition model; S4, training the constructed greenhouse recognition model using the greenhouse samples; and S5, using the trained model to extract greenhouses from the image within the target region. The advantages of this invention are: fast extraction speed and high extraction accuracy, solving the problem of the imbalance between extraction accuracy and speed and the excessive computational resource requirements of commonly used fully convolutional neural network methods.

Description

Technical Field

[0001] This invention relates to the field of remote sensing information and image processing technology, specifically to a method for extracting agricultural greenhouses based on an attention mechanism and a lightweight fully convolutional network. Background Technology

[0002] Agricultural greenhouses are common facilities in current agricultural activities. Their widespread use is due to their significant advantages: protecting crops and dramatically increasing crop yields. Greenhouses are primarily used for growing various cash crops, and understanding their quantity and distribution can indirectly estimate the economic benefits of agricultural activities. On the other hand, agriculture largely uses plastic greenhouses, and most plastics are currently non-biodegradable, causing environmental pollution if not properly disposed of. Therefore, mapping the role of greenhouses in agricultural policy, protecting the land's ecological environment, and promoting sustainable agricultural development is of great importance.

[0003] With the development of commercial drones, it is now possible to flexibly acquire high-resolution images of target areas using small multi-rotor or fixed-wing drones at a relatively low cost. However, in today's diverse data acquisition methods, the automated extraction of information from data, especially most of the information of interest in remote sensing images, still cannot meet practical needs. Currently, extraction methods for agricultural greenhouses can be divided into three main categories: 1. Manual visual interpretation; 2. Traditional machine learning methods; 3. Methods based on fully convolutional neural networks.

[0004] 1. Human visual interpretation

[0005] The manual visual interpretation method involves using a person to identify and delineate the area of ​​the greenhouse on the obtained image. Generally speaking, this method can achieve the best results, but it has the significant drawback of being time-consuming. If this method is used, the entire mission cycle will often be significantly increased, which cannot meet the needs of immediate inspection.

[0006] 2. Traditional machine learning methods

[0007] Traditional machine learning methods can be divided into two main categories: unsupervised extraction methods, which mainly rely on multispectral index calculations, but have the drawback of requiring multispectral data and only providing a qualitative reflection of greenhouse distribution; and supervised learning methods, which require "feature engineering" to design features for classification. In general, traditional machine learning methods suffer from limited accuracy and poor environmental robustness due to their inherent characteristics, failing to meet the practical needs of large-area, fast, and high-precision extraction, thus limiting their application.

[0008] 3. Methods based on deep convolutional neural networks

[0009] The development of computer vision has driven the application of artificial intelligence in remote sensing interpretation, and deep learning methods proposed in the field of computer vision have shown great potential in image recognition. Currently, fully convolutional networks in deep learning can extract images with the accuracy required for practical applications. However, fully convolutional networks have a difficult problem to overcome—the balance between accuracy, speed, and required computational resources. Using classic, high-performing models means requiring large computational resources while having a slow model training and inference extraction speed. On the other hand, using networks with fewer parameters can significantly reduce the extraction performance of high-accuracy models, potentially failing to meet practical requirements. Summary of the Invention

[0010] The purpose of this invention is to address the above-mentioned problems by providing a method for extracting agricultural greenhouses based on attention mechanisms and lightweight fully convolutional networks.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: This method for extracting agricultural greenhouse data based on attention mechanisms and lightweight fully convolutional networks includes the following steps:

[0012] S1. Acquire image data of the target area and perform data preprocessing;

[0013] S2. Create binary greenhouse samples for the target area;

[0014] S3. Construct a greenhouse identification model;

[0015] S4. Use greenhouse samples to train the constructed greenhouse identification model;

[0016] S5. Use the trained model to extract the greenhouse from the image within the target area.

[0017] The proposed model can be trained quickly and requires minimal computational resources during inference. It meets the need for rapid and accurate extraction of greenhouse areas from large-area images, addressing the imbalance between accuracy and speed, and the excessive computational demands of commonly used fully convolutional neural network methods. The invention features clear and repeatable operational steps, enabling rapid determination of greenhouse area and region, and timely information extraction.

[0018] In the above-mentioned method for extracting agricultural greenhouses based on attention mechanism and lightweight fully convolutional network, in step S1, the image of the target area is acquired by a UAV. The data preprocessing steps are successively divided into stitching, mosaicking, color balancing, orthorectification, and image cropping. Through the above steps, a complete high-resolution UAV image of the target area is obtained, and the image format is TIFF.

[0019] In the above-mentioned method for extracting agricultural greenhouses based on attention mechanisms and lightweight fully convolutional networks, step S2 specifically includes the following steps:

[0020] S21. Cropping the image of the target area selected as the sample, then outlining the greenhouse area on the cropped image and generating a surface vector file of the greenhouse area, and finally creating a label image that completely corresponds to the pixels of the area in the selected drone image.

[0021] S22. Rasterize the vector of the outlined greenhouse surface to generate a raster file corresponding to each pixel of the cropped area. Based on the annotation results, mark the greenhouse area as 1 and the non-greenhouse area as 0 in the raster file.

[0022] S23. The raster files generated from the UAV images and their annotations are cropped into corresponding small blocks, with a block size of 512*512 pixels. Finally, the generated small blocks are enhanced to generate the final sample.

[0023] The enhancement processes include: multi-scale scaling, random rotation, vertical / horizontal flipping, Gaussian blur, and random Gaussian noise.

[0024] In the above-mentioned method for extracting agricultural greenhouses based on attention mechanisms and lightweight fully convolutional networks, step S3 specifically includes the following steps:

[0025] S31 employs two 2D convolutional layers with a stride of 2 and a size of 3*3. Each convolutional layer is followed by a Batch Normalization layer and a ReLU activation function. After these two convolutional layers, the input image, which is 512*512 pixels, is downsampled to a size of 128*128 pixels, expanding the three input channels (red, green, and blue) to 24 channels. This two-layer convolutional structure reduces the resolution to decrease the size of the image to be computed, while simultaneously extracting information and expanding it to multiple channels.

[0026] S32. Use residual structures, depthwise separable convolutions, grouped convolutions, 1x1 convolutions, and dilated convolutions to construct feature encoding modules to extract information at different scales and semantic levels.

[0027] S33. Construct a decoder module using ordinary 2D convolution, 1x1 convolution and linear interpolation upsampling methods, and use this decoder module to decode features at different scales layer by layer.

[0028] S34. Construct a category attention module using self-attention. This category attention module has two output branches: a main branch that outputs the prediction result and an auxiliary branch that is only used to assist training and is not used as the prediction result for inference. When the feature map is decoded to the size of the original image, it is connected to the category attention module to complete the aggregation of contextual information.

[0029] In the above-mentioned agricultural greenhouse extraction method based on attention mechanism and lightweight fully convolutional network, in step S32, a feature encoder is constructed using three Shift Blocks and Conv Blocks. The feature map obtained after 4x downsampling through two convolutional layers is fed into the feature encoding module to obtain feature maps at different scales. The Shift Block consists of a convolutional layer + activation function layer + convolutional layer + convolutional layer + activation function layer. The Conv Block consists of a Shift Block with residual connections, that is, a connection is added between the input position of the Shift Block and the penultimate and second-to-last layers of the Shift Block to add and fuse them.

[0030] In the above-mentioned agricultural greenhouse extraction method based on attention mechanism and lightweight fully convolutional network, in step S33, a decoder module is constructed using ordinary 2D convolution, 1x1 convolution and linear interpolation upsampling methods. The decoder module decodes the feature map size to the original image size by layer-by-layer decoding of the acquired UAV image feature maps of different scales. That is, the results of Basic Conv3, Conv Block1 and Basic Conv2 are decoded layer by layer to obtain feature maps with 2 channels and four times downsampling. The Basic Conv consists of convolutional layer + normalization layer + activation function.

[0031] In the above-mentioned agricultural greenhouse extraction method based on attention mechanism and lightweight fully convolutional network, dilated convolution is used to construct a global information extraction module with dilated convolution with dilation rate of [1, 2, 5], average pooling method, and 1*1 convolution. This module calculates the results at different scales using different convolution methods on the output results, and then superimposes the calculation results for output. The output results are then subjected to 1*1 convolution to complete the aggregation of extracted information.

[0032] In the above-mentioned method for extracting agricultural greenhouses based on attention mechanism and lightweight fully convolutional network, in step S34, the obtained four-fold downsampled feature map is upsampled by four times and sent to the category attention module. In the category attention module, the input feature map is first used to generate an auxiliary branch and Query, Key, and Value through convolution of two branches. The Query and Key are multiplied to obtain the weight, and then the obtained weight is multiplied with Value to obtain the calculation result of feature map attention. Finally, the feature map attention result is superimposed with the feature map that generated the Query and sent to the last two convolutional layers to obtain the main branch output result, i.e., the prediction result.

[0033] In the above-mentioned method for extracting agricultural greenhouses based on attention mechanisms and lightweight fully convolutional networks, the training of the greenhouse recognition model in step S4 specifically includes the following steps:

[0034] S41. Batch input of paired images and tags;

[0035] S42. After network computation, the network outputs two results: the main branch prediction result and the auxiliary branch output used for training. These two outputs are first normalized across channels using the softmax method. Then, the loss function values ​​(error calculations) are performed on the normalized results and the label images respectively, and the weights of each parameter are updated using backpropagation and gradient descent.

[0036] In the above-mentioned method for extracting agricultural greenhouses based on attention mechanisms and lightweight fully convolutional networks, the loss function used is the cross-entropy loss function, with the following formula:

[0037]

[0038] Where N is the number of pixels in each image, p(x) is the true class distribution of the corresponding pixel i, and q(x) is the predicted class distribution of the corresponding pixel i.

[0039] Compared with existing technologies, the advantages of this invention are as follows: This method uses a series of computational units with small parameters to extract features, adds dilated convolution to increase the ability to extract contextual information, adopts an encoder-decoder structure to aggregate information at different scales and levels, and adds category attention during decoding to further aggregate the contextual information of the feature map. This method has the accuracy comparable to that of classic fully convolutional networks while significantly reducing the amount of computation, and solves the problem of balancing efficiency, accuracy and reliability when extracting greenhouses using fully convolutional networks. Attached Figure Description

[0040] Figure 1 This is a flowchart of the present invention;

[0041] Figure 2 This is a schematic diagram of the greenhouse extraction model in this invention;

[0042] Figure 3 This is a schematic diagram of the three basic modules in the greenhouse extraction model of the present invention;

[0043] Figure 4 This is a schematic diagram of the attention module of this invention; Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0045] like Figure 1 As shown, this method for extracting agricultural greenhouses based on attention mechanisms and lightweight fully convolutional networks includes the following steps:

[0046] S1. Acquire image data of the target area and perform data preprocessing;

[0047] S2. Create binary greenhouse samples for the target area;

[0048] S3. Construct a greenhouse identification model;

[0049] S4. Use greenhouse samples to train the constructed greenhouse identification model;

[0050] S5. Use the trained model to extract the greenhouse from the image within the target area.

[0051] This method is used to efficiently and accurately extract greenhouses from UAV images. It overcomes the shortcomings of commonly used methods, requires very low computing power for inference, and has a fast inference speed and high accuracy of inference results.

[0052] In step S1, the image of the target area is acquired by a UAV. The data preprocessing steps are stitching, mosaicking, color balancing, orthorectification, and image cropping. Through the above steps, a complete high-resolution UAV image of the target area is obtained, and the image format is TIFF.

[0053] Step S2 specifically includes the following steps:

[0054] S21. Cropping the image of the target area selected as the sample, then outlining the greenhouse area on the cropped image and generating a surface vector file of the greenhouse area, and finally creating a label image that completely corresponds to the pixels of the area in the selected drone image.

[0055] S22. Rasterize the vector of the outlined greenhouse surface to generate a raster file corresponding to each pixel of the cropped area. Based on the annotation results, mark the greenhouse area as 1 and the non-greenhouse area as 0 in the raster file.

[0056] S23. The raster files generated from the UAV images and their annotations are cropped into corresponding small blocks, each block being 512*512 pixels in size. Finally, the generated small blocks are enhanced to generate the final sample.

[0057] The enhancement processes include: multi-scale scaling, random rotation, vertical / horizontal flipping, Gaussian blur, and random Gaussian noise.

[0058] Step S2 uses the acquired and processed images to create greenhouse samples of the target area.

[0059] like Figure 2 As shown, step S3 specifically consists of the following steps:

[0060] S31 employs two 2D convolutional layers with a stride of 2 and a size of 3*3. Each convolutional layer is followed by a Batch Normalization layer and a ReLU activation function. After these two convolutional layers, the input image, which is 512*512 pixels, is downsampled to a size of 128*128 pixels, expanding the three input channels (red, green, and blue) to 24 channels. This two-layer convolutional structure reduces the resolution to decrease the size of the image to be computed, while simultaneously extracting information and expanding it to multiple channels.

[0061] S32. A feature encoding module is constructed using residual structure, depthwise separable convolution, grouped convolution, 1x1 convolution, and dilated convolution to extract information at different scales and semantic levels; the parameters of each layer are shown in Table 1.

[0062]

[0063] Table 1

[0064] S33. Construct a decoder module using ordinary 2D convolution, 1x1 convolution and linear interpolation upsampling methods, and use this decoder module to decode features at different scales layer by layer.

[0065] S34. Construct a category attention module using self-attention. This module has two output branches: a main branch that outputs the prediction result, and an auxiliary branch that is only used to assist training and not as the prediction result for inference. When the feature map is decoded back to the original image size, it is fed into the category attention module to complete the aggregation of contextual information. The category attention module is as follows: Figure 4 As shown;

[0066] Step S3 constructs the model based on the samples, generally using a fully convolutional network approach with a category attention module added at the end. The sub-networks before the category attention module are constructed using structures such as 2D convolution, residual connections, dilated convolution, 1*1 convolution, depthwise separable convolution, and grouped convolution. The overall network flow is as follows: Figure 2 As shown;

[0067] In step S32, a feature encoder is constructed using three Shift Blocks and Conv Blocks. The feature map obtained after 4x downsampling through two convolutional layers is fed into the feature encoding module to obtain feature maps at different scales. The Shift Block consists of a convolutional layer + activation function layer + convolutional layer + convolutional layer + activation function layer. The Conv Block consists of a Shift Block with residual connections, that is, a connection is added between the input position of the Shift Block and the penultimate and second-to-last layers of the Shift Block to add and fuse them.

[0068] In step S33, a decoder module is constructed using ordinary 2D convolution, 1x1 convolution, and linear interpolation upsampling methods. The decoder module decodes the feature maps of the UAV images at different scales to the original image size by layer-by-layer decoding. That is, the results of Basic Conv3, Conv Block1, and Basic Conv2 are decoded layer by layer to obtain feature maps with 2 channels and four times downsampling. The Basic Conv consists of a convolutional layer, a normalization layer, and an activation function.

[0069] Among them, dilated convolution uses dilated convolution with dilation rate of [1, 2, 5], average pooling method, and 1*1 convolution to build a global information extraction module. This module uses different convolution methods to calculate the results at different scales on the output results, and then superimposes the calculation results for output. The output results are then subjected to 1*1 convolution to complete the aggregation of extracted information.

[0070] The global information extraction module consists of Figure 2 The Basic Conv3, 4, 5, 6, and 7 in the text are used.

[0071] In step S34, the obtained feature map, which is downsampled by four times, is upsampled by four times and sent to the category attention module. In the category attention module, the input feature map is first used to generate an auxiliary branch with Query, Key, and Value through convolution of two branches. The Query and Key are multiplied to obtain the weight, and then the obtained weight is multiplied with Value to obtain the feature map attention calculation result. Finally, the feature map attention result is superimposed with the feature map that generated the Query and sent to the last two convolution layers to obtain the main branch output result, which is the prediction result.

[0072] In step S4, the training of the greenhouse identification model specifically includes the following steps:

[0073] S41. Batch input of paired images and tags;

[0074] S42. After network computation, the network outputs two results: the main branch prediction result and the auxiliary branch output used for training. The two output results are first normalized on the channel using the softmax method. Then, the loss function value is calculated for the normalized result and the label image, i.e., the error is calculated. The weight of each parameter is updated using backpropagation and gradient descent.

[0075] The loss function used is the cross-entropy loss function, and the formula is:

[0076]

[0077] Where N is the number of pixels in each image, p(x) is the true class distribution of the corresponding pixel i, and q(x) is the predicted class distribution of the corresponding pixel i.

[0078] The model is trained using the cross-entropy loss function, and backpropagation and gradient descent are used to update the network parameters to train the model.

[0079] Step S5 specifically includes the following steps:

[0080] S51. Crop and number the processed UAV image data of the target area to be extracted;

[0081] S52. Input the cropped image into the network, obtain the recognition result, name it according to the name corresponding to the input number, and save it;

[0082] S53. Reassemble the regional prediction results according to the cropped numbers to obtain the extraction results of the greenhouse in the target area.

[0083] In summary, the principle of this embodiment is as follows: a series of computational units with small parameter quantities are used to extract greenhouse features within the target area; dilated convolution is added to increase the ability to extract contextual information; an encoder-decoder structure is adopted to aggregate information at different scales and levels; and category attention is added during decoding to further aggregate the contextual information of the feature map. This method has the accuracy comparable to that of classic fully convolutional networks while significantly reducing the amount of computation, thus solving the problem of balancing efficiency, accuracy and reliability when extracting greenhouse features using fully convolutional networks.

[0084] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. Agricultural greenhouse extraction method based on attention mechanism and light-weight fully convolutional network, characterized by, The method comprises the following steps: S1, optical image data of a target area is acquired and data preprocessing is performed; S2, a two-class greenhouse sample of the target area is made; S3, a greenhouse recognition model is constructed; S31, two layers of 2D convolution with a step distance of 2 and a size of 3*3 are adopted, and a BatchNormalization layer and a ReLU activation function are matched after each convolution layer; after the two convolution layers, an image with an input size of 512*512 is down-sampled to an image with a size of 128*128, and three input channels, i.e., red, green and blue, are expanded to 24 channels; the two convolution layers reduce the resolution to reduce the size of the image to be calculated, and meanwhile, information is extracted and expanded to multiple channels; S32, a feature encoding module is constructed using a residual structure, a depth separable convolution, a grouped convolution, a 1x1 convolution and a dilated convolution, so as to extract information of different scales and different semantic levels; S33, a decoder module is constructed using a normal 2D convolution, a 1x1 convolution and a linear interpolation up-sampling method, and the decoder module is used to decode features at different scales layer by layer; S34, a class attention module is constructed using self-attention, the class attention module has two output branches, one is a main branch outputting a prediction result, and the other is an auxiliary branch used only for auxiliary training and not used as a prediction result in inference; when the feature map is decoded to the original image size, the class attention module is accessed to complete context information aggregation; S4, the constructed greenhouse recognition model is trained using the greenhouse sample; S41, a batch of paired images and labels is input; S42, after network calculation, two results are output at the end of the network, one is a main branch prediction result, and the other is an auxiliary branch output used for auxiliary training; firstly, the two output results are normalized on the channel using a softmax method, and then the normalized results and label images are used to calculate loss function values, i.e., to calculate errors, and the weights of each parameter are updated using back propagation and gradient descent method; The loss function used is a cross-entropy loss function, and the formula is: Wherein, N is the number of pixels of each image, p(x) is the real class distribution of the corresponding pixel point i, and q(x) is the predicted class distribution of the corresponding pixel point i; S5, the trained model is used to extract greenhouse images in the target area.

2. The agricultural greenhouse extraction method based on attention mechanism and light-weight fully convolutional network according to claim 1, characterized in that, In step S1, the image of the target area is acquired by a UAV, and the data preprocessing steps are sequentially divided into splicing, inlaying, color uniformity, orthographic correction, image cropping, and the complete high-resolution UAV image of the target area is obtained through the above steps, and the image format is TIFF. 3.The agricultural greenhouse extraction method based on attention mechanism and light-weight fully convolutional network according to claim 1, characterized in that, Step S2 is specifically divided into the following steps: S21, the image of the target area selected as a sample is cropped, and then the greenhouse area is outlined on the cropped image to generate a face vector file of the greenhouse area, and finally a label image corresponding to the pixels of the selected area in the UAV image is made; S22, vector rasterization is performed on the outlined greenhouse surface, a raster file corresponding to the cutting area pixel by pixel is generated, and according to the labeling result, the greenhouse area is marked as 1 and the non-greenhouse area is marked as 0 in the raster file; S23, the unmanned aerial vehicle image and the raster file generated by labeling are cut into small blocks corresponding to each other, the size of the small blocks is 512*512 pixels, and finally the generated small blocks are enhanced to generate the final sample; The enhancement processing includes: multi-scale scaling, random rotation, vertical / horizontal flipping, Gaussian blur, and random Gaussian noise.

4. The agricultural greenhouse extraction method based on attention mechanism and light-weight fully convolutional network according to claim 3, characterized in that, In step S32, a feature encoder is constructed using three Shift Blocks and Conv Blocks, and the feature map obtained by 4 times down-sampling through two layers of convolution is input into the feature encoding module to obtain feature maps at different scales; the Shift Block is composed of a convolution layer + an activation function layer + a convolution layer + a convolution layer + an activation function layer, and the Conv Block is composed of a Shift Block with a residual connection, that is, a connection is added between the input position of the Shift Block and the second layer from the last of the Shift Block for addition fusion. 5.The agricultural greenhouse extraction method based on attention mechanism and light-weight fully convolutional network according to claim 4, characterized in that, In step S33, a decoder module is constructed using ordinary 2D convolution, 1x1 convolution and linear interpolation up-sampling method, the decoder module uses layer-by-layer decoding to decode the feature map size to the original size, that is, the results of Basic Conv3, Conv Block1 and Basic Conv2 are decoded layer by layer to obtain a feature map with a channel number of 2 and a four times down-sampling, wherein the Basic Conv is composed of a convolution layer + a normalization layer + an activation function. 6.The agricultural greenhouse extraction method based on attention mechanism and light-weight fully convolutional network according to claim 5, characterized in that, The dilated convolution uses dilated convolution with an expansion rate of [1, 2, 5], average pooling method, and 1*1 convolution to construct a global information extraction module, which calculates the results at different scales using different convolution methods and then outputs the calculation results, and then performs 1*1 convolution on the output results to complete the convergence of the extracted information.

7. The agricultural greenhouse extraction method based on attention mechanism and light-weight fully convolutional network according to claim 6, characterized in that, In step S34, the four times down-sampled feature map is up-sampled by four times, and is input into a category attention module, in which the input feature map is first used to generate an auxiliary branch and Query, Key, Value using convolution of two branches, wherein the Query and the Key are multiplied to obtain a weight, then the obtained weight and the Value are multiplied to obtain the calculation result of the feature map attention, finally the feature map attention result and the feature map generated by the Query are superimposed and input into the last two convolution layers to obtain the output result of the main branch, that is, the prediction result. 8.The agricultural greenhouse extraction method based on attention mechanism and light-weight fully convolutional network according to claim 7, characterized in that, Step S5 specifically includes the following steps: S51, the unmanned aerial vehicle image data of the target region to be extracted is processed, cut and numbered; S52, the cut image is input into the network to obtain the recognition result, which is named and saved according to the corresponding name of the input number; S53, the region prediction result is reassembled according to the cutting number to obtain the extraction result of the target region greenhouse.

Images