Agricultural greenhouse rapid extraction method based on lightweight fully convolutional neural network

Abstract

The scheme discloses a kind of agricultural greenhouse fast extraction methods based on lightweight fully convolutional neural network, the method uses the way of different inflation rate's convolution structure and selectively decoding to encoding information to the global information and local information are aggregated, fuse the information under different receptive field;And in encoder, decoder, all completely use depth separable convolution to reduce the redundancy of convolutional neural network calculation.This solves the technical problems existing in the prior art, such as long cycle, low precision or the need for a large number of computing resources, has the advantages of fast and accurate extraction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing and information extraction, and particularly relates to a method for rapidly extracting agricultural greenhouses based on a lightweight fully convolutional neural network architecture. BACKGROUND

[0002] Agricultural greenhouses are common cultivation facilities in modern facilities. Since the use of the first generation of greenhouses, the area of global agricultural greenhouses has reached 301.9 million mu. On the one hand, it can extend the growing season, protect food crops from adverse growing conditions, and significantly improve crop yields, and is considered one of the important facilities for the evolutionary transition from extensive agriculture to intensive agriculture. On the other hand, since the materials used by most greenhouses are difficult to be completely degraded by the natural environment, it has caused the rapid expansion of related environmental problems, such as soil pollution, plastic waste, biological degradation, and farmland destruction. Therefore, mapping and estimating the coverage of greenhouses is crucial for the sustainable development of regional agriculture and environment, but the extraction method commonly used by manually sketching on unmanned aerial vehicle images is very time-consuming and laborious, and the sketching time accounts for 70-80% of the entire process.

[0003] Thanks to the development of commercial unmanned aerial vehicles, it can obtain high-resolution data at relatively low cost and near real time. At present, unmanned aerial vehicle remote sensing has become one of the important remote sensing methods. On the other hand, the development of convolutional neural networks has also promoted the intelligent transformation of unmanned aerial vehicle image interpretation methods, however, convolutional neural networks usually require a lot of computing resources, so there is an urgent need for efficient and accurate lightweight networks to extract greenhouses.

[0004] The most commonly used agricultural greenhouse extraction method at present mainly includes four types: 1) visual interpretation method; 2) multi-spectral index calculation; 3) supervised classification method based on pixels / object-oriented; 4) deep convolutional neural network-based method.

[0005] 1) Visual interpretation method

[0006] Among them, the visual interpretation method is to manually search for agricultural greenhouses in the image and manually sketch, and the accuracy of this method depends on professional knowledge and experience. Although professional training can usually achieve good accuracy, if this method is used, the time occupied by this method can reach 70-80% of the entire task, which cannot meet the demand of automatic large-scale extraction.

[0007] 2) Multi-spectral index calculation

[0008] The multispectral index calculation method takes a multispectral image or a hyperspectral image with multiple bands as a data source, and at least five bands of red, green, blue, near-infrared, and short-wave infrared are needed. However, although the model based on the spectral index can be used for rapid large-scale mapping of multispectral satellites, there are the following significant disadvantages in using this method: a. If fine mapping of agricultural greenhouses is required, a multi-spectral sensor needs to be carried by a UAV to obtain the required data source, which greatly increases the cost. b. The spectrum will change due to different materials and use time of the greenhouse, so there will be an error requirement that does not meet the actual demand in actual detection. c. The method of using spectrum calculation will cause inaccurate boundary capture due to boundary mixed pixels, noise, and actual imaging environment. Therefore, this method is more suitable for qualitative analysis and is not suitable for accurate mapping of agricultural greenhouses.

[0009] 3) Supervised classification method based on pixel / object-oriented

[0010] This method can be classified as a traditional machine learning method. The pixel-based method is performed on a single pixel, only considering the different bands of spectral characteristics, so this method will cause "salt and pepper noise". The object-oriented method classifies objects composed of multiple adjacent homogeneous pixels. This method not only uses spectral information, but also considers the texture and shape of the agricultural greenhouse. This method can effectively avoid the salt and pepper noise generated by the pixel-based method. However, this method needs to manually design and filter features, that is, analyze their relationship and select the best feature combination. The method of manually designing features results in relatively shallow features, thus limiting the accuracy.

[0011] 4) Method based on deep convolutional neural network

[0012] Convolutional neural networks are a technology developed in recent years. Deep learning methods can automatically and efficiently extract various levels of features from input data. In the case of given labels, the area of the greenhouse can be mapped with an accuracy that completely surpasses various traditional methods. However, using convolutional neural networks faces a major problem: convolutional neural networks are usually composed of millions of parameters, so a large amount of computing resources is needed. This makes the calculation speed not meet the actual demand when used on a large scale. SUMMARY

[0013] The purpose of the present application is to solve the above problems, and to provide a lightweight full convolutional neural network-based method for rapid extraction of agricultural greenhouses. The specific scheme is as follows:

[0014] A lightweight full convolutional neural network-based method for rapid extraction of agricultural greenhouses,

[0015] The network structure of the greenhouse lightweight model is as follows:

[0016] Two layers of 2D convolution are used to process the original input image of three channels of red, green and blue, and output a feature map;

[0017] The tenth convolution structure Basic Conv10 is used to decode the feature map;

[0018] The feature map is processed by three layers of encoding, and the output of the first layer of encoding is decoded using the eighth convolution structure Basic Conv8;

[0019] The third convolution structure Basic Conv3, the fourth convolution structure Basic Conv4, the fifth convolution structure Basic Conv5 and the sixth convolution structure Basic Conv6 with different expansion rates are used to process the output results of the three layers of encoding;

[0020] The seventh convolution structure Basic Conv7 is used to converge and abstract the outputs of the third convolution structure Basic Conv3, the fourth convolution structure Basic Conv4, the fifth convolution structure Basic Conv5 and the sixth convolution structure Basic Conv6;

[0021] The outputs of the eighth convolution structure Basic Conv8 and the seventh convolution structure Basic Conv7 are spliced as the input of the ninth convolution structure Basic Conv9;

[0022] The outputs of the ninth convolution structure Basic Conv9 and the tenth convolution structure Basic Conv10 are spliced, and the spliced results are processed by the eleventh convolution structure Basic Conv11 and the twelfth convolution structure Basic Conv12 to obtain a four times down-sampled feature map;

[0023] The four times down-sampled feature map is up-sampled by four times to obtain a prediction map consistent with the size of the input image;

[0024] A greenhouse lightweight model is used:

[0025] The prediction map and the label image are calculated by a loss function, and the weights are updated by backpropagation and gradient descent method to train the greenhouse lightweight model;

[0026] The trained greenhouse lightweight model is used to output recognition results based on input images.

[0027] In the above method for quickly extracting agricultural greenhouses based on a lightweight fully convolutional neural network, two layers of 2D convolution are used to down-sample and encode the input image, so that the size of the output feature map is reduced while the high-frequency and low-frequency information in the image is transmitted to different encoding channels.

[0028] In the above method for quickly extracting agricultural greenhouse based on lightweight fully convolutional neural network, the step distance of the two 2D convolutions is 2, and four times down-sampling is performed while obtaining information, reducing the size of the feature map;

[0029] And the two 2D convolutions are the first convolution structure Basic Conv1 and the second convolution structure Basic Conv2 respectively with BN layer and ReLU activation function.

[0030] In the above method for quickly extracting agricultural greenhouse based on lightweight fully convolutional neural network, the first layer of the two 2D convolutions reduces the resolution of the input image to 1 / 2 and expands the feature information to 16 channels; the second layer further reduces the resolution of the first layer by 1 / 2 and encodes the feature information to 24 channels.

[0031] In the above method for quickly extracting agricultural greenhouse based on lightweight fully convolutional neural network, the feature map is processed by three substructures composed of Shift Block and Conv Block from input to output;

[0032] As shown in Figure 3b , the Shift Block includes convolution layer, ReLU activation function, convolution layer, convolution layer, ReLU activation function from input to output in turn;

[0033] As shown in Figure 3c , the Conv Block includes convolution layer, ReLU activation function, convolution layer, convolution layer, ReLU activation function from input to output in turn, and the input of the first convolution layer in the structure is connected with the output of the third convolution layer in the residual after the ReLU activation function.

[0034] In the above method for quickly extracting agricultural greenhouse based on lightweight fully convolutional neural network, the output of the seventh convolution structure Basic Conv7 is twice up-sampled and spliced with the output of the eighth convolution structure Basic Conv8 as the input of the ninth convolution structure Basic Conv9;

[0035] The output of the ninth convolution structure Basic Conv9 is twice up-sampled and spliced with the output of the tenth convolution structure Basic Conv10;

[0036] The eleventh convolutional structure Basic Conv11 and the twelfth convolutional structure Basic Conv12 process the splicing result of the output of the ninth convolutional structure Basic Conv9 and the output of the tenth convolutional structure Basic Conv10 to obtain a feature map with a channel number of 2 and four times of down-sampling.

[0037] In the above method for quickly extracting an agricultural greenhouse based on a lightweight fully convolutional neural network, before a loss function is calculated for a prediction map and a label image, the prediction map is normalized on two channels using a softmax method, and a class with a maximum prediction value is taken as a current pixel point class.

[0038] In the above method for quickly extracting an agricultural greenhouse based on a lightweight fully convolutional neural network, the input image is an unmanned aerial vehicle image obtained through unmanned aerial vehicle remote sensing technology.

[0039] Sample data for training a greenhouse lightweight model is obtained in the following manner:

[0040] A1. Obtain an unmanned aerial vehicle image of a target area, and perform preprocessing including splicing, inlaying, color smoothing, orthographic correction, and image cropping to obtain a high-resolution unmanned aerial vehicle image TIFF map of the target area;

[0041] A2. Make a two-class greenhouse sample, select a greenhouse in a part of the area of the unmanned aerial vehicle image, and manually draw the boundary to make a label image corresponding to the pixels of the selected area in the unmanned aerial vehicle image;

[0042] A3. Vectorize the drawn boundary to generate a raster file corresponding to the pixels of the cropped area, and according to the annotation result, mark the greenhouse area as 1 and the non-greenhouse area as 0 in the raster file;

[0043] A4. Crop the unmanned aerial vehicle image and the raster generated by the annotation into small blocks corresponding to each other, with a size of 512*512 pixels;

[0044] A5. Perform enhancement processing on the small blocks generated in A4 to generate the final sample data.

[0045] In the above method for quickly extracting an agricultural greenhouse based on a lightweight fully convolutional neural network, in step A5, the enhancement processing includes any one or more of multi-scale scaling, rotation, flipping, blurring, and random noise.

[0046] In the above method for quickly extracting an agricultural greenhouse based on a lightweight fully convolutional neural network, the method for outputting a recognition result based on an input image using the trained greenhouse lightweight model is specifically as follows:

[0047] B1. The unmanned aerial vehicle image of the target area is preprocessed, and is cropped to the size required by the greenhouse lightweight model input and is numbered in sequence;

[0048] B2. The cropped small block is input into the network to obtain the identification result and is named in a manner corresponding to the input number;

[0049] B3. The results of the region prediction are spliced according to the cropped number to obtain the identification result.

[0050] The advantages of the present application are that the agricultural greenhouse extraction method has the characteristics of rapidness and accuracy, and the global information and local information are aggregated and the information under different receptive fields is fused by using convolution structures (Basic Conv3, 4, 5, 6) with different expansion rates and selectively decoding the encoding information. Figure 6 As shown in the figure, the network proposed in the present application is about 1 / 8 to 1 / 10 of the size of the commonly used network, and the IoU intersection ratio (precision) of the greenhouse is basically consistent with the U-Net++ model, so it can be seen that the method significantly reduces the requirement for computing resources in the actual production environment while ensuring high precision. The technical problems of long cycle (visual interpretation), low precision (traditional method) or requiring a large amount of computing resources (mainstream fully convolutional neural network) existing in various mainstream methods are solved. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The flowchart of the agricultural greenhouse identification and extraction of the agricultural greenhouse rapid extraction method based on the lightweight fully convolutional neural network disclosed in the embodiment of the present application;

[0052] Figure 2 The schematic diagram of the model constructed by the agricultural greenhouse rapid extraction method based on the lightweight fully convolutional neural network disclosed in the embodiment of the present application;

[0053] Figure 3a 、 Figure 3b 、 Figure 3c The schematic diagram of the Basic Conv, Shift Block and Conv Block of the greenhouse identification lightweight model disclosed in the embodiment of the present application;

[0054] Figure 4 The method example of splicing the middle part of each prediction map disclosed in the embodiment of the present application;

[0055] Figure 5 The partial result of the greenhouse extraction in Jiaxing City on the unmanned aerial vehicle image by using the present application;

[0056] Figure 6Model size, IoU intersection over union (precision) comparison for various feature extractors for semantic segmentation. DETAILED DESCRIPTION

[0057] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0058] The scheme is based on a full convolutional neural network and is used for accurately extracting a greenhouse on a UAV image. The method has a significantly higher extraction accuracy than commonly used exponential, pixel-based and object-oriented methods, and requires less computing power than commonly used full convolutional neural network methods, has a faster prediction speed, and has the advantages of not requiring high platform performance.

[0059] Figure 1 The flowchart of the agricultural greenhouse rapid extraction method based on the lightweight deep convolutional neural network architecture disclosed in the embodiment of the application is shown in Figure 1 The agricultural greenhouse extraction method based on the UAV image comprises the following steps:

[0060] Step one, data preprocessing of the target area UAV image, including splicing, inlaying, color uniformity, orthographic correction, image cropping, etc., finally obtaining the complete high-resolution UAV image TIFF of the target area.

[0061] Step two, making a greenhouse sample:

[0062] (1) Selecting a part of the greenhouse in the region of the UAV image and manually drawing the boundary, finally making a label map corresponding to the pixels of the selected region in the UAV image;

[0063] (2) Vector rasterization of the drawn boundary, generating a raster file corresponding to the cropped region pixel by pixel, and according to the annotation result, marking the greenhouse region as 1 and the non-greenhouse region as 0 in the raster file;

[0064] (3) Cropping the UAV image and the raster file generated by the annotation into small blocks corresponding to each other, with a size of 512*512 pixels;

[0065] (4) Finally, the small blocks generated in (3) are enhanced to generate the final sample. The enhancement includes: multi-scale scaling, rotation, flipping, blurring, random noise;

[0066] Step three, constructing a greenhouse lightweight model, using a full convolutional architecture, and using residual connection, hollow convolution, 1x1 convolution, group convolution, etc. to design a lightweight greenhouse lightweight model, the overall network process is as follows:

[0067] (1) First, two layers of 2D convolution are used, i.e., a first convolution structure Basic Conv1 and a second convolution structure Basic Conv2. The two layers are substructures for resampling and encoding the input image in the greenhouse extraction network. After the two layers of down-sampling and encoding, the size of the output feature map is reduced, and the high-frequency and low-frequency information in the image is transmitted to different encoding channels. The structure of Basic Conv is shown in FIG. 8. The model of the present scheme involves 12 Basic Convs, and the Conv parameters of each Basic Conv are shown in Table 1. The first layer of the two layers of Basic Conv reduces the resolution of the input image to 1 / 2 of the input image and expands the feature information to 16 channels. The second layer further reduces the resolution of the result of the first layer by 1 / 2 and encodes the feature map to 24 channels. Figure 3a

[0068] (2) The feature map obtained in (1) is automatically extracted and encoded at different scales to obtain multiple feature maps at multiple scales. Specifically, the first feature map is output by the second convolution structure Basic Conv2, and the first feature map is decoded by the tenth convolution structure Basic Conv10. The first feature map is processed by three layers of encoding to obtain the third feature map. The second feature map is output by the first layer of encoding, and the second feature map is decoded by the eighth convolution structure Basic Conv8. The third convolution structure Basic Conv3, the fourth convolution structure Basic Conv4, the fifth convolution structure Basic Conv5, and the sixth convolution structure Basic Conv6 with different inflation rates are used to process the third feature map processed by three layers of encoding to obtain the fourth, fifth, sixth, and seventh feature maps.

[0069] The three-layer encoding structure includes three layers of structures each composed of a Shift Block and a Conv Block. The present scheme involves 3 groups of Shift Blocks and 3 groups of Conv Blocks, and the Conv parameters of each group are shown in Table 1.

[0070] (3) The obtained unmanned aerial vehicle images at different scales are decoded layer by layer to decode the results of the encoder. More specifically, the results of Basic Conv3, 4, 5, and 6 are spliced and passed through Basic Conv7 and two times up-sampling. The results of the previous step are spliced with the results of Basic Conv8, and then passed through Basic Conv9. The output of Basic Conv9 is up-sampled by two times and spliced with the results of Basic Conv10. Finally, after passing through Basic Conv11 and 12 convolution layers, a feature map with a channel number of 2 and four times down-sampling is obtained. ​

[0071] (4) The four times down-sampled feature map obtained in (3) is up-sampled by four times to obtain a prediction map with two channels and consistent with the size of the input image;

[0072] (5) After the obtained feature map consistent with the input size is normalized on two channels using the softmax method, the class with the maximum class prediction value is taken as the current pixel point class, i.e., it is predicted as a greenhouse or a non-greenhouse.

[0073] Step four, greenhouse identification. The trained greenhouse lightweight model is used to extract and identify the greenhouses in the target area from the UAV images of the target area. This includes:

[0074] (1) The processed UAV images of the target area to be extracted are cut into the required size according to a certain overlap and numbered in order;

[0075] (2) The cut small blocks are input into the network to obtain the identification results and named in a manner corresponding to the input number;

[0076] (3) The results of the region prediction are reassembled according to the cutting number. Because the cutting is sliding cutting according to a certain overlap, only the middle part of each small block is taken when splicing, as shown in Figure 4 The final identification result of the greenhouse is the result after splicing.

[0077] Table 1. Parameter settings of each convolutional layer of the greenhouse lightweight model.

[0078]

[0079] The specific embodiments described in the present embodiment are only illustrative of the spirit of the present application. Those skilled in the art to which the present application belongs can make various modifications or supplements to the described specific embodiments or replace them with similar ways, without deviating from the spirit of the present application or exceeding the scope defined by the appended claims.

Claims

1. A method for rapid extraction of agricultural greenhouse based on lightweight fully convolutional neural network, characterized in that, The application relates to a greenhouse lightweight model and a training method thereof. The network structure for constructing the greenhouse lightweight model comprises the following steps: Two layers of 2D convolution are adopted to input the image for processing, and feature maps are outputted; The tenth convolution structure Basic Conv10 is used to decode the feature maps; The feature maps are processed through three layers of encoding, and the eighth convolution structure Basic Conv8 is used to decode the output of the first layer of encoding; The third convolution structure Basic Conv3, the fourth convolution structure Basic Conv4, the fifth convolution structure Basic Conv5 and the sixth convolution structure Basic Conv6 with different expansion rates are used to process the output results of the three layers of encoding respectively; The seventh convolution structure Basic Conv7 is used to converge and abstract the outputs of the third convolution structure Basic Conv3, the fourth convolution structure Basic Conv4, the fifth convolution structure Basic Conv5 and the sixth convolution structure Basic Conv6; The output of the eighth convolution structure Basic Conv8 and the seventh convolution structure Basic Conv7 is spliced to serve as the input of the ninth convolution structure Basic Conv9; The output of the ninth convolution structure Basic Conv9 and the output of the tenth convolution structure Basic Conv10 are spliced, and the spliced result is processed through the eleventh convolution structure Basic Conv11 and the twelfth convolution structure Basic Conv12 to obtain a four times down-sampled feature map; The four times down-sampled feature map is up-sampled by four times to obtain a prediction map with the same size as the input image; The greenhouse lightweight model is used to: The prediction map and a label image are subjected to loss function calculation, and the weight is updated by using the back propagation and gradient descent method to train the greenhouse lightweight model; The greenhouse lightweight model is used to output an identification result based on an input image.

2. The method of claim 1, wherein the method is based on a lightweight fully convolutional neural network. Two layers of 2D convolution are used to down-sample and encode the input image, so that the size of the output feature map is reduced, and the high-frequency and low-frequency information in the image is transmitted to different encoding channels.

3. The method of claim 2, wherein the method is based on a lightweight fully convolutional neural network. The step distance of the two layers of 2D convolution is 2; The first convolution structure Basic Conv1 and the second convolution structure Basic Conv2 are respectively the first convolution structure Basic Conv1 and the second convolution structure Basic Conv2 which are respectively provided with a BN layer and a RELU activation function.

4. The method of claim 3, wherein the method is based on a lightweight fully convolutional neural network. The first layer of the two layers of 2D convolution reduces the resolution of the input image to 1 / 2 and expands the feature information to 16 channels; the second layer further reduces the resolution of the result of the first layer to 1 / 2 and encodes the feature information to 24 channels.

5. The method of claim 1, wherein the method is based on a lightweight fully convolutional neural network. The feature maps are processed through three layers of encoding by using three substructures which are respectively composed of a Shift Block and a Conv Block from input to output; The Shift Block comprises, from input to output, a convolution layer, a ReLU activation function, a convolution layer, a convolution layer and a ReLU activation function; The Conv Block comprises, in sequence from input to output, a convolutional layer, a ReLU activation function, a convolutional layer, a convolutional layer, and a ReLU activation function, and the input of the first convolutional layer is connected in residual manner with the output of the third convolutional layer and then input to the second ReLU activation function.

6. The method of claim 1, wherein the method is based on a lightweight fully convolutional neural network. The output of the seventh convolutional structure Basic Conv7 is spliced with the output of the eighth convolutional structure Basic Conv8 after being up-sampled by a factor of two to serve as the input of the ninth convolutional structure Basic Conv9; The output of the ninth convolutional structure Basic Conv9 is spliced with the output of the tenth convolutional structure Basic Conv10 after being up-sampled by a factor of two; The eleventh convolutional structure Basic Conv11 and the twelfth convolutional structure Basic Conv12 process the spliced result of the output of the ninth convolutional structure Basic Conv9 and the output of the tenth convolutional structure Basic Conv10 to obtain a feature map with a channel number of 2 and a down-sampling factor of four.

7. The method of claim 6, wherein the method is based on a lightweight fully convolutional neural network. Before the loss function calculation is performed on the prediction map and the label image, the prediction map is normalized on two channels using a softmax method, and the class with the maximum prediction value is taken as the class of the current pixel point.

8. The method of claim 1, wherein the method is based on a lightweight fully convolutional neural network. The input image is an unmanned aerial vehicle image obtained through unmanned aerial vehicle remote sensing technology. The sample data for training the greenhouse lightweight model is obtained in the following manner: A1. Obtain an unmanned aerial vehicle image of a target region, and perform preprocessing including splicing, inlaying, color uniformization, orthographic correction, and image cropping to obtain a high-resolution unmanned aerial vehicle image TIFF map of the target region; A2. Create a two-class greenhouse sample, select a greenhouse in a part of the region in the unmanned aerial vehicle image, and manually draw the boundary to create a label image corresponding to the pixels in the selected region in the unmanned aerial vehicle image; A3. Vectorize the drawn boundary to generate a raster file corresponding to the pixels in the cropped region, and according to the labeling result, mark the greenhouse region as 1 and the non-greenhouse region as 0 in the raster file; A4. Crop the unmanned aerial vehicle image and the generated raster into small blocks corresponding to each other, each with a size of 512*512 pixels; A5. Perform enhancement processing on the small blocks generated in A4 to generate the final sample data.

9. The method of claim 8, wherein the method is based on a lightweight fully convolutional neural network. In step A5, the enhancement processing includes any one or more of multi-scale scaling, rotation, flipping, blurring, and random noise.

10. The method of claim 8, wherein the method is based on a lightweight fully convolutional neural network. The method for outputting a recognition result based on an input image using the trained greenhouse lightweight model specifically comprises: B1. Preprocess the unmanned aerial vehicle image of the target region, and crop it to the size required by the greenhouse lightweight model and number it in order; B2. Input the cropped small blocks into the network to obtain the recognition result and name it in a manner corresponding to the input number; B3. Re-splice the results of the region prediction according to the cropping numbers to obtain the recognition result.

Images