A method for real-time detection of satellite remote sensing images of farmland

Abstract

The application discloses a kind of real-time detection farmland satellite remote sensing image method, comprising the following steps: obtaining farmland satellite remote sensing image dataset;Hybrid gate linear feedforward neural network and double-path high-low frequency feature extraction module are constructed, SegFormer network is improved, and farmland remote sensing image segmentation model is obtained;The farmland remote sensing image segmentation model is trained based on the farmland satellite remote sensing image dataset;Based on the farmland remote sensing image segmentation model after training, the farmland satellite remote sensing image to be processed is segmented and handled.The application can better carry out feature extraction and identification to real remote sensing farmland image, even in complex remote sensing image, also can more accurately identify the same kind of ground object with greater difference, distinguish similar different farmland, greatly improve the segmentation precision of farmland remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of farmland satellite remote sensing image detection, and particularly relates to a method for real-time detection of farmland satellite remote sensing images. BACKGROUND

[0002] The traditional farmland statistical classification method is mainly manual field investigation and measurement. The staff records the information such as the position, area and planted crop types of the farmland through field visits, relying on simple measuring tools (such as a tape measure) and naked eye observation. This method has many shortcomings. First, the efficiency is extremely low, and a large area of farmland statistics requires a large amount of manpower and time. Second, the accuracy is insufficient, and manual measurement is prone to errors, and the estimation of the area of irregularly shaped farmland may have a large deviation. Third, the information is not updated in time, and due to the high cost and long cycle of manual statistics, it is difficult to keep up with the speed of changes in farmland use, such as rapid adjustment of crop planting structure. Moreover, this method is greatly affected by subjective factors, and different staff may have different understandings of the farmland classification standard, resulting in inconsistent classification results. The use of satellite remote sensing data for farmland statistical classification has become a research hotspot due to its large detection range and high timeliness, and in recent research, image segmentation methods based on deep learning provide a new way to improve the ability of remote sensing data to accurately segment farmland images.

[0003] However, in the task of farmland satellite remote sensing image classification, the farmland images have a large intra-class gap and high similarity between different classes, and common semantic segmentation methods have certain deficiencies in distinguishing different types of farmland and identifying the boundaries of ridges and roads. The general transform model and cnn model perform unsatisfactorily in work.

[0004] Therefore, in order to more accurately identify the same ground objects with large differences and distinguish similar different farmlands in complex remote sensing images, a method for real-time detection of farmland satellite remote sensing images based on a Segformer model is proposed. SUMMARY

[0005] To solve the above technical problems, the present application provides a method for real-time detection of farmland satellite remote sensing images to solve the problems existing in the prior art.

[0006] To achieve the above purpose, the present application provides a method for real-time detection of farmland satellite remote sensing images, comprising the following steps:

[0007] Obtaining a farmland satellite remote sensing image dataset;

[0008] The mixed gate linear feedforward neural network and the dual-path high-low frequency feature extraction module are constructed to improve the SegFormer network to obtain a farmland remote sensing image segmentation model; wherein the mixed gate linear feedforward neural network is obtained by adding a multilayer perceptron layer and a gated channel attention mechanism in a mixed feedforward network;

[0009] The farmland remote sensing image segmentation model is trained based on the farmland satellite remote sensing image dataset;

[0010] The farmland satellite remote sensing image to be processed is segmented based on the trained farmland remote sensing image segmentation model.

[0011] Optionally, before training the farmland remote sensing image segmentation model based on the farmland satellite remote sensing image dataset, the method further comprises:

[0012] The farmland satellite remote sensing image dataset is subjected to random horizontal flip, random transposition and random scaling data enhancement processing.

[0013] Optionally, the formula of the mixed gate linear feedforward neural network is as follows:

[0014] X OUTPUT =MLP(GELU(DeepConv 3X3 (MLP(X in ))·MLP(X in ))+X i

[0015] Wherein, MLP is a multilayer perceptron layer, DeepConv is a deep convolution, GELU is an activation function, and X in is an input.

[0016] Optionally, the dual-path high-low frequency feature extraction module comprises a high-frequency feature extraction unit and a low-frequency feature extraction unit for high-frequency feature extraction and low-frequency feature extraction of the farmland satellite remote sensing image, respectively.

[0017] Optionally, the formula of the high-frequency feature extraction unit is as follows:

[0018]

[0019] Wherein, Q is a query vector, K is a key vector, V is a value vector, D is the total dimension of Q, K and V vectors, and SoftMax represents an activation function.

[0020] Optionally, the formula of the low-frequency feature extraction unit is as follows:

[0021]

[0022] wherein avg represents average pooling.

[0023] Optionally, the intersection over union, the average intersection over union, the average precision, and the MF1 are further included as evaluation indexes, and the farmland remote sensing image segmentation model is evaluated.

[0024] The application further provides a computer device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0025] The application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method.

[0026] The application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the method.

[0027] Compared with the prior art, the application has the following advantages and technical effects:

[0028] In the prior art, although the farmland monitoring task based on remote sensing satellite remote sensing images is increasingly mature, there are still problems such as a large intra-class difference of farmland images and a high similarity of different farmland images, which affect the accuracy of the task. Compared with the above method, the method provided by the application improves the SegFormer network by constructing a hybrid gated linear feed-forward neural network and a double-path high-low frequency feature extraction module to obtain a farmland remote sensing image segmentation model, and then performs segmentation processing on the farmland satellite remote sensing image to be processed based on the trained farmland remote sensing image segmentation model. The application can better extract and identify the features of real remote sensing farmland images based on computer vision, and even in complex remote sensing images, it can more accurately identify the same ground objects with large differences and distinguish similar different farmlands, greatly improving the segmentation accuracy of farmland remote sensing images. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their

[0030] Figure 1 The equation flowchart is an embodiment of the application;

[0031] Figure 2 The hybrid gated linear feed-forward neural network is a schematic diagram of an embodiment of the application;

[0032] Figure 3 The double-path high-low frequency feature extraction module is a schematic diagram of an embodiment of the application;

[0033] Figure 4 A farmland remote sensing image segmentation model structure diagram of an embodiment of the present application;

[0034] Figure 5 A segmentation effect comparison diagram of a LoveDA test data set using a general segmentation model of an embodiment of the present application;

[0035] Figure 6 A segmentation effect comparison diagram of a Barley Remote Sensing test data set using a general segmentation network of an embodiment of the present application. DETAILED DESCRIPTION

[0036] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0037] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0038] Embodiment one

[0039] As shown in the accompanying drawings, the present embodiment provides a method for real-time detection of farmland satellite remote sensing images, comprising the following steps: Figure 1

[0040] Obtaining a farmland satellite remote sensing image data set;

[0041] Constructing a hybrid gate linear feedforward neural network and a double-path high-low frequency feature extraction module to improve the SegFormer network and obtain a farmland remote sensing image segmentation model; wherein the hybrid gate linear feedforward neural network is obtained by adding a multilayer perceptron layer and a gated channel attention mechanism in a hybrid feedforward network;

[0042] Training the farmland remote sensing image segmentation model based on the farmland satellite remote sensing image data set;

[0043] Segmenting the farmland satellite remote sensing image to be processed based on the trained farmland remote sensing image segmentation model.

[0044] As an implementable way, the process of obtaining a farmland satellite remote sensing image data set comprises:

[0045] ​The public dataset Barley Remote Sensensing provided by the "County Agricultural Brain AI Challenge" of Ali Tianchi and the LoveDA dataset proposed by the RSIDEA team of Wuhan University are selected for experiments.

[0046] The LoveDA dataset is a land cover classification dataset proposed by the RSIDEA team of Wuhan University, which contains 5987 0.3m high-resolution images from Wuhan, Nanjing and Changzhou. The resolution of each image is 1024x1024. The dataset includes 7 land cover categories: building, road, water, wasteland, forest, agriculture and background. Specifically, 2522 images are used for training, 1669 for validation and 1796 for testing.

[0047] Barley Remote Sensing is a high-resolution remote sensing image of crops in Xingren City, Guizhou Province, taken by a drone in 2019. The spectrum is the visible light band (RGB). The dataset divides the training samples into 4 categories: barley (label 1), corn (label 2), flue-cured tobacco (label 3) and others (label 0). The dataset consists of 4 large-sized remote sensing high-resolution images with an average size of 1.8GB. The size ranges from 18576x68363 pixels to 55128x49447 pixels, which cannot be directly used as input to neural networks. Therefore, the image needs to be cropped into multiple 512x512 pixel sub-images. After cropping, the dataset has a total of 11362 sub-images, of which the training set contains 7953 images, the validation set contains 2272 images, and the test set contains 1137 images. The ratio of the training set and the validation set to the test set is about 7 to 2 to 1.

[0048] This embodiment unifies the input size of the two datasets for training and testing to 512x512 through cropping, and uses random horizontal flipping, random transposition and random scaling (scaling ratio from 0.5 to 2.0) as the data augmentation strategy.

[0049] As an implementable way, the process of constructing a hybrid gated linear feedforward neural network and a dual-path high-low frequency feature extraction module to improve the SegFormer network to obtain a farmland remote sensing image segmentation model includes:

[0050] The embodiment takes the Segformer network as the basic network, and the SegFormer network is composed of an encoder and a decoder. The Tansformer module in the encoder first uses an overlap patch embedding (OPE) module to slice the input picture, and extracts features and down-samples the tensor of the sliced picture. Because the OPE module can overlap between slices, the elements within the slice will also be connected to ensure the continuity and integrity of the elements. The obtained slice features are input into the efficient multihead self-attention (EMSA) layer and the mixed feed forward (MixFFN) layer to construct global connections. To replace the position encoding in the ordinary Transformer module, the SegFormer network adds a 3*3 two-dimensional convolution layer between the two linear transformation layers of the feedforward layer to obtain the position information in the picture space. Each Transformer module uses multiple stacked EMSA and MixFFN modules to deepen the network depth to extract more rich detail and semantic features. By using different sizes of slices, the picture can be self-attention calculated in the Transformer module at each scale. Compared with the method of gradually deepening the receptive field of the convolutional neural network, the SegFormer network can more quickly obtain global information.

[0051] In order to better improve the segmentation accuracy of satellite remote sensing images of farmland, the embodiment constructs a mixed gated linear feedforward neural network and a double-path high-low frequency feature extraction module to improve the SegFormer network:

[0052] (1) Mixed gated linear feedforward neural network (MIX-Gated Linear Feedforward Network):

[0053] The remote sensing farmland image has a large intra-class gap and a high similarity between different classes of farmland. In order to solve this problem, a multi-layer perceptron layer (MLP) is added to the mixed feedforward neural network, and a gated channel attention mechanism is added. The gated channel attention mechanism can help the network better adapt to the feature distribution of different input data, improve the generalization ability of the model, and reduce the risk of overfitting, as shown in Figure 2

[0054] ​Each token in the mixed gated linear feedforward neural network (MIX-Gated Linear Feedforward Network, MGLFFD) has a unique gating signal based on its recent fine-grained features, which solves the problem of the global average pooling being too coarse-grained in the SE mechanism. The mixed feedforward neural network does not have a position encoder design, and provides position information through deep convolution, which can effectively enhance the feature expression ability of the model and improve the attention and understanding ability of the feedforward neural network to the input data, thereby further optimizing the performance and effect of the neural network. And the numerical branch of this design has the same depth as the mixed feedforward neural network, making it more suitable for back propagation. These characteristics make the mixed gated linear feedforward neural network a simple but more powerful FFN layer, meeting the diverse needs of remote sensing image segmentation.

[0055] The mixed gated linear feedforward neural network can be represented as:

[0056] X OUTPUT =MLP(GELU(DeepConv 3X3 (MLP(X in ))·MLP(X in ))+X i ,

[0057] Wherein, MLP is a multi-layer perceptron layer, DeepConv is a deep convolution, GELU is an activation function, and X in is the input.

[0058] (2) Dual-Path High-Low Frequency Feature Extraction Module (DPHL):

[0059] Farmland remote sensing images usually have a wide geographical range, and the image is large. In addition, the image also needs to capture the detailed information of small crops, land use types and vegetation, etc. Therefore, it is necessary to process large-size images to obtain global information, while also needing to accurately extract and analyze subtle features. In order to capture more abstract and deep features in the image, the Transformer self-attention mechanism may cause the network to focus too much on high-level, abstract features, ignoring the details of the image, such as texture, edges, etc. This situation may cause the loss of key detailed information, thereby affecting the overall understanding of the image, or causing difficulties in positioning and identifying the edge contours of local targets.

[0060] To solve the above problems and enhance the model's ability to acquire low-scale information from farmland remote sensing images, the embodiment designs a dual-path high-low frequency feature extraction module (DPHL). DPHL processes high-frequency interactions through relatively high-frequency feature mapping and local self-attention encoding, processes low-frequency interactions through global attention encoding, and combines down-sampling feature mapping. The original remote sensing image has rich and diverse frequencies, among which high frequencies and low frequencies have different effects on image encoding. High frequencies are mainly used to extract local details, and low frequencies are mainly used to extract overall structure and features.

[0061] Specifically, DPHL includes a high-frequency feature extraction unit and a low-frequency feature extraction unit for extracting high-frequency and low-frequency features using two attention paths. As shown in Figure 3 In the upper path, the embodiment assigns several heads to the high-frequency feature extraction unit to capture fine-grained high frequencies through an efficient self-attention mechanism. The low-frequency feature extraction path first applies average pooling to each feature map to obtain low-frequency signals. Then, the remaining heads are assigned to the low-frequency feature extraction unit to simulate the relationship between each query position in the input feature map and the average pool low-frequency key and value of each window. Benefiting from the reduction in the length of the key and value, the low-frequency feature extraction unit also realizes a significant reduction in complexity. Finally, the high-frequency features and low-frequency features are output separately, and the results are forwarded to the subsequent layers.

[0062] Further, for each attention head, the multi-head attention uses different linear transformations to generate corresponding Q, K, and V, denoted as where are the weight matrices of the i-th head, respectively.

[0063] Further, the high-frequency feature extraction unit can be represented as:

[0064]

[0065] where Q is the query vector, which represents the query information of the current position and is used to find related information in other positions, K is the key vector, which is the feature representation of other positions and is used for similarity calculation with the Q vector, and V is the value vector, which contains the actual content or feature value of each position. The Q, K, and V vectors obtained by linear transformation of the input have a total dimension represented by "D", while the dimension of each head is "d", and the relationship "D = h * d" is satisfied, where "h" is the number of heads, and SoftMax is an activation function.

[0066] Further, the low-frequency feature extraction unit can be represented as:

[0067]

[0068] wherein avg represents average pooling.

[0069] (3) Based on the mixed gate linear feedforward neural network and the dual-path high-low frequency feature extraction module, the SegFormer network is improved to obtain a farmland remote sensing image semantic segmentation network T_SegFormer, i.e. a farmland remote sensing image segmentation model. The model structure is as shown in Figure 4 The network extracts and fuses different features through the dual-path high-low frequency feature extraction module (DPHL), thereby improving the segmentation performance of the network.

[0070] As an implementable manner, the experimental platform of the embodiment is WIN10 operating system, the CPU is ntel(R) Xeon(R) Gold 6136 CPU@3.00GHz 2.99GHz, the GPU is NVIDIA RTX 3090, and the display memory is 24GB. The development environment is Python 3.8.0, PyTorch 1.10.1, and CUDA 11.3. The Adamw optimizer is used during training, the initial learning rate is 0.0006, the decay rate is 0.01, the Batch size is set to 4, and the loss function adopts the cross-entropy loss function (CELoss). The training count adopts iterations, i.e. iterating once for each batch size of data, and the embodiment sets it to 160000 times.

[0071] As an implementable manner, the evaluation index of the comparative test of the embodiment is:

[0072] The embodiment adopts four commonly used image segmentation model indexes, i.e. the intersec-tion overunion (IoU), the mean intersection overunion (MIoU), the Mean Accuracy (mAcc), and the MF1 (mean F1 score), to evaluate the performance of the model under different categories.

[0073] wherein:

[0074]

[0075] P is a predicted region (predicted bounding box or segmentation result), G is a real region (real label), and |·| represents the area or size of the region.

[0076]

[0077] N is the number of categories, IoU i is the IoU value of the i-th category.

[0078]

[0079] where C is the number of categories, TP i is the number of true positives (the number of pixels correctly classified by the model as the i-th category), FP i is the number of false positives (the number of pixels incorrectly classified by the model as the i-th category).

[0080]

[0081] Precision (precision) is Recall (recall) is TP, FP, FN are the number of true positives, false positives and false negatives, respectively.

[0082] As an implementable way, the embodiment preliminarily verifies the segmentation performance of the improved Segformer network:

[0083] Select the Segformer of MiT-B2, swin-UperNet, DeepLabV3+, FCN and other models for comparison on two public datasets. In order to ensure the fairness of the experiment and the reliability of the data, all networks are trained and tested under the same hardware and software environment. Table 1 is the comparison of general segmentation networks on LoveDA dataset, and Table 2 is the comparison of general segmentation networks on Barley RemoteSensing dataset.

[0084] Table 1

[0085]

[0086]

[0087] Table 2

[0088]

[0089] From the experimental data, the performance of segformer is better, so the proposed method is based on the segformer model.

[0090] As an implementable way, the embodiment verifies the farmland remote sensing image segmentation model improved based on the Segformer network:

[0091] To verify the segmentation performance of the improved model, the embodiment selects the Segformer of MiT-B2, swin-UperNet, DeepLabV3+, FCN, RSSForm and other models to compare with T-Segformer on two public data sets. In order to ensure the fairness of the experiment and the reliability of the data, all networks are trained and tested under the same software and hardware environment. The bold font in the table indicates the optimal data in each index, and the data in the table with underline is the sub-optimal data in each index. Table 3 is the comparison of T-Segformer and general segmentation network on LoveDA data set, and table 4 is the comparison of T-Segformer and general segmentation network on Barley Remote Sensing data set.

[0092] Table 3

[0093]

[0094]

[0095] Table 4

[0096]

[0097] Ablation experiment:

[0098] In order to verify the effectiveness of each module of T_segformer, the module ablation experiment of the algorithm model is carried out on the two data sets of LoveDA and Barley Remote Sensing to prove the effectiveness of the module. The experiment uses mIou and MF1 two evaluation indexes to evaluate the segmentation performance of the network. The original ablation experiment results of using MiT-B2 are shown in table 5.

[0099] Table 5

[0100]

[0101] In Table 5, after adding the MGLFFD module in the Baseline network, the mIou index of the model on the LoveDA and Barley RemoteSensing two datasets is improved by 1.64% and 1.72% respectively, and the data improvement shows that improving the feedforward neural response network is effective, and the improved FFN layer enhances the feature extraction capability of the model and improves the segmentation performance of the network; after the Baseline uses the DPHL MGLFFD module, the mIou index of the model on the two datasets is improved by 1.13% and 2.44% respectively, which proves that the DPHL module can better extract the shallow information of the remote sensing image and improve the segmentation performance of the network. Finally, after adding all the modules in the Baseline network, the best segmentation performance is achieved. The experimental results prove the necessity and effectiveness of each key module of T_segformer.

[0102] Embodiment Two

[0103] The embodiment also provides a computer device comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0104] Embodiment Three

[0105] The embodiment also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method.

[0106] Embodiment Four

[0107] The embodiment also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the method.

[0108] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting satellite remote sensing images of farmland in real time, characterized in that, The method comprises the following steps: obtaining a farmland satellite remote sensing image dataset; constructing a hybrid gate linear feedforward neural network and a double-path high-low frequency feature extraction module to improve the SegFormer network and obtain a farmland remote sensing image segmentation model; wherein the hybrid gate linear feedforward neural network is obtained by adding a multilayer perceptron layer and a gated channel attention mechanism in a hybrid feedforward network; training the farmland remote sensing image segmentation model based on the farmland satellite remote sensing image dataset; segmenting the farmland satellite remote sensing image to be processed based on the trained farmland remote sensing image segmentation model; The formula of the hybrid gate linear feedforward neural network is as follows: where MLP is a multi-layer perceptron layer, is a depth convolution, GELU is an activation function, is an input; The double-path high-low frequency feature extraction module comprises a high-frequency feature extraction unit and a low-frequency feature extraction unit, and is used for high-frequency feature extraction and low-frequency feature extraction of the farmland satellite remote sensing image, respectively.

2. The method of claim 1, wherein, before training the farmland remote sensing image segmentation model based on the farmland satellite remote sensing image dataset, the method further comprises: performing random horizontal flip, random transposition and random scaling data enhancement processing on the farmland satellite remote sensing image dataset.

3. The method of claim 1, wherein, The formula of the high-frequency feature extraction unit is as follows: where Q is a query vector, K is a key vector, V is a value vector, D is the total dimension of Q, K, V vectors, denotes an activation function.

4. The method of claim 3, wherein, The formula of the low-frequency feature extraction unit is as follows: wherein avg represents average pooling.

5. The method of claim 1, wherein, further comprising evaluating the farmland remote sensing image segmentation model by taking intersection over union, mean intersection over union, mean accuracy and MF1 as evaluation indexes.

6. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 5. The processor executes the computer program to realize the steps of the method of any one of claims 1-5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1-5.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1-5. The computer program is executed by the processor to realize the steps of the method of any one of claims 1-5.

Images