A spatiotemporal semantic segmentation method for agricultural land cover based on Transformers-MulMLA

Abstract

This invention provides a spatiotemporal semantic segmentation method for agricultural land cover based on Transformers-MulMLA, belonging to the technical field of spatiotemporal semantic segmentation of agricultural land cover. The technical problem to be solved is to provide an improvement to the spatiotemporal semantic segmentation method for agricultural land cover based on Transformers-MulMLA. The technical solution adopted to solve the above technical problem includes the following steps: S1: Image preprocessing of the input crop image at time t; S2: Inputting the image processed in step S1 into a spatial feature extraction Transformer encoder to output spatial features; S3: Inputting the spatial features in step S2 into a temporal feature extraction Transformer encoder for temporal analysis to output time series features; S4: Decoding the coding layer features using a Mul-MLA structure to obtain a semantic segmentation image of the target crop cover distribution. This invention is applied to agricultural land segmentation.

Description

Technical Field

[0001] The invention provides a Transformers-MulMLA-based agricultural land cover spatiotemporal semantic segmentation method, belonging to the technical field of agricultural land semantic segmentation methods. Background Art

[0002] Drawing crop distribution maps is a key step in realizing many smart agriculture applications. Analyzing the crop categories and crop areas in crop distribution maps can provide key data support for yield prediction, farming guidance and environmental assessment, and provide decision-making basis for studying regional food balance, predicting agricultural regional productivity and population carrying capacity. Therefore, drawing efficient and accurate crop distribution maps is particularly important.

[0003] At present, the research and application in the field of land use analysis and crop cover analysis can be mainly regarded as the semantic segmentation problem of spatial and temporal features. On the one hand, different types of crops have different spectral distribution characteristics in remote sensing satellite images. On the other hand, different types of crops generally have their own unique temporal variation characteristics. The existing technology classifies each pixel on the remote sensing satellite image according to the above characteristics of the crops, thereby realizing the semantic segmentation of satellite remote sensing images and identifying the distribution of different types of crops in the image.

[0004] At present, most technologies only analyze one of the spatial features and temporal features. For example, U-Net technology uses convolutional neural networks to form an encoder-decoder structure, downsamples and upsamples images, and uses a skip connection structure to perform feature splicing and fusion, thereby achieving image semantic segmentation. However, when processing temporal information, U-Net has difficulty learning the correlation of feature distributions of different temporal sequences. Another example is Bi-LSTM (Long short-term memory network) technology, which often reflects its advantages in the recurrent neural network structure in the field of analyzing temporal features, but the low attention to spatial correlation means that the technology ignores the positional relationship of crops in satellite remote sensing images, resulting in a high pixel-level classification error rate.

[0005] Therefore, the modeling and analysis of spatiotemporal features is the key to the semantic segmentation of crop cover distribution. For example, 3D-CNN technology uses the convolutional neural network structure to share the convolution kernel to generate multiple feature maps for continuous spatial features in the time dimension, so that the feature maps have temporal correlation, thereby achieving a certain degree of synchronous modeling and analysis of spatiotemporal features. However, due to the limitations of convolutional neural network technology, this method can only perform feature analysis on several adjacent frames of temporal information and cannot trace the long-term temporal feature change law. ConvLSTM technology expands the fully connected LSTM (FC-LSTM) to have a convolutional structure in both input-to-state and state-to-state transitions, so that it can better capture spatiotemporal correlation. However, this type of method also inherits the network defects of the LSTM structure. First, the long sequence dependency and gradient vanishing problems make it impossible to accurately learn features for long time series problems. Second, the parallel stacking structure of ConvLSTM limits the propagation of temporal information between layers. The STATT structure combines Unet with LSTM structures, using the Unet encoder-decoder structure to capture image spatial features, and using the Bi-LSTM structure to capture temporal features between images, thereby realizing the learning and analysis of spatiotemporal information. The attention mechanism helps select temporal information for importance ranking, and time nodes that have a greater impact on crop coverage classification will be assigned greater weights for focused learning, which to a certain extent solves the long-term dependency problem and gradient vanishing problem of the LSTM structure when drawing crop distribution maps. Although this method can accurately segment different crops and geographical modules in terms of image segmentation, it cannot accurately classify crops in the segmented areas in terms of semantic segmentation, resulting in serious defects in its learning efficiency and prediction efficiency. The reason is that the U-Net structure does not learn the similarities and differences in the appearance of crops in the analysis of the spatial features of large-resolution images such as satellite cloud images.

[0006] To sum up, based on the above-mentioned industry status and the problems existing in various technologies in this field, in order to perform a more complete, accurate and detailed feature analysis of spatiotemporal features, to solve the defects of the prior art in the simultaneous analysis of temporal and spatial features, and to solve the problem of difficulty in pixel-level semantic segmentation of remote sensing satellite images, the present invention proposes a spatiotemporal agricultural land cover information semantic segmentation method based on Transformers-MulMLA. Summary of the invention

[0007] In order to overcome the deficiencies in the prior art, the present invention aims to solve the technical problem of providing an improved method for spatiotemporal semantic segmentation of agricultural land cover based on Transformers-MulMLA.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for spatiotemporal semantic segmentation of agricultural land cover based on Transformers-MulMLA, comprising the following steps:

[0009] S1: Perform image preprocessing on the input crop image at time t;

[0010] S2: Input the image processed in step S1 into the spatial feature extraction Transformer encoder, perform spatial feature analysis and output spatial features;

[0011] S3: Input the spatial features in step S2 into the temporal feature extraction Transformer encoder for temporal analysis and output the time series features;

[0012] S4: The Mul-MLA structure is used to decode the coding layer features to obtain the semantic segmentation image of the target crop cover distribution.

[0013] The step S1 of performing image preprocessing on the crop image is specifically as follows:

[0014] Assume that the input image size is H*W*C, where H and W are the length and height of the image, and C is the number of channels. The image is divided into feature maps of size P*P*C, then the number of blocks is N=H*W / (P*P), and then the small feature maps are linearly mapped into a one-dimensional vector, then the total input transformation of the input image is N*(P 2 *C);

[0015] Next, position embedding is performed to learn the relative positions of pixels. Specifically, the image features after the above size transformation are accumulated with a learnable randomly initialized position code and input into the spatial feature extraction Transformer encoder.

[0016] The spatial feature extraction Transformer encoder used to extract spatial features in step S2 is L layers, that is, it includes L layers of multi-head self-attention mechanisms and multi-layer MLP modules. In each layer, the input of the multi-head self-attention layer is (Q, K, V) (Query, Key, Value). At time t, the input of the lth layer is W Q , W K , W V are the weight parameter matrices of Q, K, and V respectively. The calculation formula of Multihead self-attention is:

[0017]

[0018]

[0019] Where d is the scaling factor, and its specific value is the dimension of Q, W O Compute parameter matrix for multi-head attention;

[0020] After the residual link and MLP module, the output of the first layer Transformer encoder is:

[0021]

[0022] Finally, the output feature of the spatial feature analysis Transformer encoder at time t is L is the number of Transformer encoder layers.

[0023] The step S3 is specifically as follows:

[0024] The Transformer encoder performs temporal analysis on the spatial features of the previous stage. Assuming that the length of the time series is T, the spatial features of the input image at each time scale are extracted. It is easy to obtain the time series feature Z = [Z 1 , Z 2 , ..., Z T ], input the time series features into the 12-layer Transformer encoder, and obtain the encoding layer feature H = [H 1 , H 2 , ..., H 12 ].

[0025] The step S4 is specifically as follows:

[0026] The Mul-MLA structure is used to decode the coding layer features. The Mul-MLA structure adopts a bidirectional feature aggregation strategy to process the selected coding layer features from top to bottom and from bottom to top respectively.

[0027] First, the encoding layer output feature H = [H 1 , H 2 , ..., H 12 ] is restored to P*P*C, and the restored features are divided into four groups, namely G 1 =[H 1 , H 5 , H 9 ]; G 2 =[H 2 , H 6 , H 10 ]; G 3 =[H 3 , H 7 , H 11 ]; G 4 =[H 4 , H8 , H 12 ]. 1 , G 3 From top to bottom (from H 9 , H 12 Start aggregating features) Each layer passes through a 1*1 convolution and a 3*3 convolution, and then uses deconvolution to upsample to obtain the output feature O 1 =[O 1 , O 5 , O 9 ]、O 3 =[O 3 , O 7 , O 11 ]; at the same time G 2 , G 4 From bottom to top (from H 2 , H 4 Each layer takes the same operation, through a 1*1 convolution and a 3*3 convolution, and then uses deconvolution to upsample to obtain the output feature. 2 =[O 2 , O 6 , O 10 ]、O 4 =[O 4 , O 8 , O 12 ];

[0028] Secondly, after concatenating the above output features, the upsampled features are smoothed through a 3*3 convolutional layer and a 1*1 convolutional layer to finally obtain a semantic segmentation image of the target crop cover distribution.

[0029] The upsampling is specifically performed using 4*4 deconvolution, 16*16 deconvolution, batch normalization and ReLU operations.

[0030] The beneficial effects of the present invention compared with the prior art are as follows:

[0031] Compared with the prior art, a large number of experiments have proved that the Transformers spatiotemporal feature information mining technology adopted in the present invention has stronger spatial feature analysis capabilities than the existing CNN architecture network model when the sample size is sufficient. The reason is that the Transformer structure can use the global sub-attention model to learn the global feature information of the image sequence to obtain the long-range dependency information in the image, thereby avoiding the problem that the CNN network structure gradually reduces the spatial resolution and cannot learn the long-range dependency information of the image scene due to the limited receptive field.

[0032] In terms of time series feature analysis capabilities, the present invention solves the long time series dependency problem and gradient vanishing problem inherent in the LSTM network structure. At the same time, the Transformer structure is more efficient and easier to train than the LSTM network structure.

[0033] In terms of decoding, the Mul-MLA decoder innovatively adopted in the present invention performs bidirectional feature aggregation on the features after timing analysis. Compared with the existing MLA and Bi-MLA methods which only select part of the coding layer output features for decoding operations, Mul-MLA decodes all the coding layer output features without losing the timing and spatial feature information obtained by the coding layer analysis. It can also effectively avoid the bias of traditional decoders towards the feature information of previous time nodes and better learn the correlation of global timing feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention will be further described below in conjunction with the accompanying drawings:

[0035] Figure 1 It is a schematic diagram of the structure of the Transformers-MulMLA model structure framework of the present invention;

[0036] Figure 2 It is a schematic diagram of the MLA Block structure of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the invention is described in detail below with reference to the accompanying drawings. Figure 1 .

[0038] The present invention proposes a semantic segmentation method for spatiotemporal information of agricultural land cover based on Transformer-MulMLA. 1. This method first uses Transformer technology as a spatial feature information encoder to extract spatial feature information from agricultural satellite cloud images.

[0039] 2. Secondly, this method uses Transformer technology again to extract the temporal information features of the spatial feature information of different time series extracted by multiple Transformer encoders.

[0040] 3. Finally, this method uses Mul-MLA technology as a feature information decoder, and realizes a comprehensive analysis of time series information through the bidirectional feature aggregation characteristics of Mul-MLA technology, and finally realizes high-precision semantic segmentation of agricultural land cover spatiotemporal information, and realizes pixel-level classification and recognition of different types of crop areas in agricultural satellite cloud maps.

[0041] The specific steps of the present invention are as follows:

[0042] (1) The present invention first analyzes the spatial features of the input image at time t, assuming that the input image size is H*W*C, where H and W are the length and height of the image, and C is the number of channels. It is divided into feature maps of size P*P*C, and the number of blocks is N=H*W / (P*P). Then the small feature maps are linearly mapped into a one-dimensional vector, and the total input transformation of the input image is N*(P 2 *C). Next, position embedding is performed to learn the relative positions of pixels and input into the spatial feature extraction Transformer encoder.

[0043] (2) As an encoder-decoder structure based on multi-head self-attention, Transformer has not only demonstrated excellent performance in capturing temporal features in recent years, but also demonstrated better modeling capabilities than convolutional neural networks in capturing image spatial features.

[0044] The Transformer encoder used to extract spatial features in the present invention includes L layers of multi-head self-attention mechanisms and multi-layer perceptron (MLP) modules. In each layer, the input of the multi-head self-attention layer is (Q, K, V), i.e. (Query, Key, Value). The calculation process of multi-head self-attention (SA) is:

[0045]

[0046]

[0047] After the residual link and MLP module, the output of the Transformer encoder of this layer is:

[0048]

[0049] Finally, the output features of the spatial feature analysis Transformer encoder at time t can be described as

[0050] (3) Secondly, the second Transformer encoder for extracting time series features in the present invention will perform time series analysis on the spatial features of the previous stage. Assuming that the time series length is T, the spatial features of the input image at each time scale are extracted, and the time series feature Z = [Z 1 , Z 2 , ..., Z T ]. This feature will be input into the 12-layer Transformer encoder to obtain the encoding layer feature H = [H 1 , H2 , ..., H 12 ].

[0051] (4) Finally, the Mul-MLA structure is used to decode the coding layer features. This structure adopts a bidirectional feature aggregation strategy to process the selected coding layer features from top to bottom and from bottom to top. First, the coding layer output feature H = [H 1 , H 2 , ..., H 12 ] is restored to P*P*C, and the restored features are divided into four groups, namely G 1 =[H 1 , H 5 , H 9 ]; G 2 =[H 2 , H 6 , H 10 ]; G 3 =[H 3 , H 7 , H 11 ]; G 4 =[H 4 , H 8 , H 12 ]. 1 , G 3 From top to bottom (from H 9 , H 12 Start aggregating features) Each layer passes through a 1*1 convolution and a 3*3 convolution, and then uses deconvolution to upsample to obtain the output feature O 1 =[O 1 , O 5 , O 9 ]、O 3 =[O 3 , O 7 , O 11 ]; at the same time G 2 , G 4 From bottom to top (from H 2 , H 4 Each layer takes the same operation, through a 1*1 convolution and a 3*3 convolution, and then uses deconvolution to upsample to obtain the output feature. 2 =[O 2 , O 6 , O 10 ]、O 4 =[O 4 , O 8 , O 12], the upsampling is performed by 4*4 deconvolution, 16*16 deconvolution, batch normalization and ReLU operations. Secondly, the above output features are concatenated and smoothed through a 3*3 convolution layer and a 1*1 convolution layer to finally obtain the semantic segmentation image of the target crop cover distribution.

[0052] What needs to be explained about the specific structure of the present invention is that the connection relationship between the various component modules adopted in the present invention is definite and feasible. Except for the special instructions in the embodiments, the specific connection relationship can bring about the corresponding technical effects, and solve the technical problems raised by the present invention without relying on the execution of the corresponding software program. Except for the specific instructions, the models and connection methods of the components, modules and specific components appearing in the present invention belong to the existing technologies such as published patents, published journal articles, or common knowledge that can be obtained by technical personnel in this field before the application date, and there is no need to elaborate, so that the technical solution provided in this case is clear, complete and feasible, and the corresponding physical products can be reproduced or obtained according to the technical means.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for spatiotemporal semantic segmentation of agricultural land cover based on Transformers-MulMLA, characterized by: The steps include: S1: Perform image preprocessing on the input crop image at time t; S2: Input the image processed in step S1 into the spatial feature extraction Transformer encoder, perform spatial feature analysis and output spatial features; S3: Input the spatial features in step S2 into the temporal feature extraction Transformer encoder for temporal analysis and output the time series features; S4: using the Mul-MLA structure to decode the coding layer features to obtain a semantic segmentation image of target crop cover distribution; the step S4 is specifically as follows: The Mul-MLA structure is used to decode the coding layer features. The Mul-MLA structure adopts a bidirectional feature aggregation strategy to process the selected coding layer features from top to bottom and from bottom to top respectively. First, select some encoding layer features H = [H 1 , H 2 , ..., H 12 ], restore the selected feature shape to P*P*C, and divide the restored features into four groups, namely G 1 =[H 1 , H 5 , H 9 ]; G 2 =[H 2 , H 6 , H 10 ]; G 3 =[H 3 , H 7 , H 11 ]; G 4 =[H 4 , H 8 , H 12 ]; Among them G 1 Since H 9 Start to H 1 Aggregate features, G 3 Since H 11 Start to H 3 Aggregate features, each layer passes through a 1*1 convolution and a 3*3 convolution, and then uses deconvolution to upsample to obtain the output feature O 1 =[O 1 , O 5 , O 9 ]、O 3 =[O 3 , O 7 , O 11 ]; at the same time G 2 Since H 2 Start to H 10 Aggregate features, G 4 Since H 4 Start to H 12 Aggregate features, each layer takes the same operation, through a 1*1 convolution and a 3*3 convolution, and then uses deconvolution to upsample to obtain the output feature O 2 =[O 2 , O 6 , O 10 ]、O 4 =[O 4 , O 8 , O 12 ]; Secondly, after concatenating the above output features, the upsampled features are smoothed through a 3*3 convolutional layer and a 1*1 convolutional layer to finally obtain a semantic segmentation image of the target crop cover distribution.

2. The method for spatiotemporal semantic segmentation of agricultural land cover based on Transformers-MulMLA according to claim 1, characterized in that: The specific steps of performing image preprocessing on the crop images in step S1 are as follows: Assume that the input image size is H'*W*C, where H' and W are the length and height of the image, and C is the number of channels. The image is divided into feature maps of size P*P*C, then the number of blocks is N=H'*W / (P*P), and then the small feature maps are linearly mapped into a one-dimensional vector, then the total input transformation of the input image is N*(P 2 *C); Next, position embedding is performed to learn the relative position of pixels and input into the spatial feature extraction Transformer encoder.

3. The method for spatiotemporal semantic segmentation of agricultural land cover based on Transformers-MulMLA according to claim 2, characterized in that: The spatial feature extraction Transformer encoder for extracting spatial features in step S2 includes L layers of multi-head self-attention layers and multi-layer MLP modules. The Transformer encoder has L layers in total. In each layer of the Transformer encoder, the input of the multi-head self-attention layer is (Q, K, V) or (Query, Key, Value). In the first layer of the Transformer encoder, 1≤L, the input W Q , W K , W V are the weight parameter matrices of Q, K, and V respectively. The calculation formula of Multihead self-attention is: d is the scale factor, W O Compute parameter matrix for multi-head attention; After the residual link and MLP module, the output of the first layer Transformer encoder is: Finally, the output feature of the spatial feature analysis Transformer encoder at time t is L is the number of Transformer encoder layers.

4. The method for spatiotemporal semantic segmentation of agricultural land cover based on Transformers-MulMLA according to claim 3, characterized in that: The step S3 is specifically as follows: The Transformer encoder performs temporal analysis on the spatial features of the previous stage. Assuming that the length of the time series is T', the input image of each time scale is subjected to spatial feature extraction to obtain the time series feature Z = [Z 1 , Z 2 , ..., Z T’ ], input the time series features into the L-layer Transformer encoder, and obtain the encoding layer feature set H = [H 1 , H 2 , ..., H L ].

5. The method for spatiotemporal semantic segmentation of agricultural land cover based on Transformers-MulMLA according to claim 1, characterized in that: The upsampling is specifically performed using 4*4 deconvolution, 16*16 deconvolution, batch normalization and ReLU operations.

Images