Maskform cultivated land parcel identification method based on multi-scale query

Abstract

The invention provides a Maskform cultivated land parcel identification method based on multi-scale query, and the method comprises the steps: constructing an MSQform model based on a Maskform model, the MSQform model comprises a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module and a segmentation prediction module, training the MSQform model based on a preset data set, obtaining the trained MSQform model, carrying out the segmentation prediction of the multi-scale feature extraction and embedding module, the mask generation module, the multi-scale mask fusion module and the segmentation prediction module, and carrying out the segmentation prediction of the multi-scale feature extraction and embedding module, the multi-scale feature extraction and embedding module, the multi-scale feature extraction and embedding module and the multi-scale mask fusion module. And carrying out cultivated land parcel identification based on the trained MSQform model. According to the method, the accuracy of farmland block mask prediction is improved by means of multiple queries, and meanwhile, the farmland block mask prediction is carried out based on each query and the feature map of the corresponding scale, so that the prediction precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of farmland plot recognition, and in particular to a Maskformer farmland plot recognition method based on multi-scale query. Background Art

[0002] As the basic unit of farmland, cultivated land is the key to farmland information extraction, which is very important for agricultural information survey, agricultural development and yield estimation, and planting pattern promotion. With the increasing availability of high-resolution remote sensing images, research on automated extraction of cultivated land boundaries has received favorable support. However, due to the influence of different imaging conditions, complex background features, and changing time stages, it is still a challenge to intelligently obtain high-precision farmland boundaries based on high-resolution remote sensing images.

[0003] Deep neural networks have made significant breakthroughs in many computer vision tasks. Semantic segmentation networks such as U-net and Deeplab can provide pixel-by-pixel predictions and are applied to farmland extraction tasks. Most farmland extraction works based on deep learning combine semantic segmentation with edge detection to improve the detection accuracy of plot boundaries. Existing methods are usually based on convolutional neural networks, which can automatically learn high-level deep features implicitly from training data through local connections, weight sharing, and sampling. However, such methods often lack the ability to capture long-range dependencies, and their extraction performance is limited in a wider range of applications. Since the application of ViT in the field of computer vision, Transformer-based models have achieved extraordinary achievements with superior long-range modeling capabilities, becoming a new standard paradigm for semantic segmentation. On this basis, some people have improved the Transformer method and applied it to farmland extraction tasks, providing a research reference for improving the accuracy of farmland extraction in complex scenes. Considering that cultivated land objects are closed and continuous areas, mask classification based methods may be a new idea. One of the representative models for mask classification based on Tranformer is Maskformer, which is particularly suitable for processing image features with large intra-class variance and small inter-class variance in remote sensing scenes. Mask classification is to predict a set of binary masks and assign a single category to each mask.

[0004] However, there are still some problems in applying Maskformer to farmland extraction. First, Maskformer generates masks through downsampled feature maps, while farmland extraction often faces multi-scale effects and a large number of farmland, which may cause the complex contours of farmland blocks to be ignored or become ambiguous. Second, Maskformer performs semantic segmentation on farmland tasks with a single category but a large number of objects. In essence, it merges all farmland blocks into one object prediction mask. It is difficult to find all farmland blocks accurately through a single query vector, which inevitably causes some plots to be missed and merged. Therefore, it is very necessary to design a Maskformer farmland block recognition method based on multi-scale queries. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the object of the present invention is to provide a Maskformer farmland plot recognition method based on multi-scale query.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The present invention provides a Maskformer cultivated land plot identification method based on multi-scale query, comprising:

[0008] The MSQformer model is constructed based on the Maskformer model, wherein the MSQformer model includes a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module and a segmentation prediction module;

[0009] The MSQformer model is trained based on a preset data set to obtain a trained MSQformer model;

[0010] Cultivated land plots are identified based on the trained MSQformer model.

[0011] Preferably, the multi-scale feature extraction and embedding module is used to extract multi-scale features through the backbone network, and input the final feature map as the embedded feature into the transformer decoder for decoding, specifically:

[0012] The collected image RGB∈R H×W Input the backbone network ResNet50 for feature extraction to obtain the reduced resolution feature map scale Among them, C F is the channel dimension of the image feature, S i =2 3+i(i=1,2,3,4), and input it into Transformer decoder for semantic decoding. Transformer Decoder is composed of several stacked Transformer decoder layers, and different numbers of query vectors are generated for the same Transformer Decoder. To generate hidden features of corresponding scale Among them C Q is the query vector dimension, N i is the number of query vectors, C H is the hidden layer feature dimension, i = 1, 2, 3, and finally the hidden layer features and image features with one-to-one correspondence at multiple scales are obtained.

[0013] Preferably, the mask generation module is used to generate a prediction mask according to the hidden layer features and influencing features at each scale, specifically:

[0014] The image is divided into two regions and a binary mask {m i |m i ∈[0, 1] H×W}, where i = 0, 1 indicates that the category is background or farmland. All farmland objects are included in a mask as a whole area. The hidden layer features contain the global mask information of each category. The hidden layer features and image features are respectively passed through the fully connected layer and the convolution layer to obtain the mask embedding vector and pixel embedding vector Multiply the two together to get the prediction Among them, S i =2 3+i And i=1,2,3.

[0015] Preferably, the multi-scale mask fusion module is used to fuse multi-scale prediction masks, specifically:

[0016] Based on the multi-scale mask fusion module, different scales are gradually fused Where S i =2 3+i And i = 1, 2, 3, where the low-resolution mask is fused with the higher-resolution mask through a layer of convolution and upsampling operations, and so on to complete the fusion of masks of different resolutions at three scales to obtain finer plots and boundaries, and finally Input segmentation prediction module for segmentation.

[0017] Preferably, the segmentation prediction module is used to perform segmentation prediction according to the fused mask, specifically:

[0018] Hidden features of the third scale based on Transformer decoder output Generate a category prediction vector through a fully connected layer Among them, K is the number of categories, and the prediction result is obtained by performing a dot multiplication operation with the fused mask and activating it with the sigmoid function. The predicted mask is upsampled to the original resolution size to obtain the segmentation result seg∈R K×H×W .

[0019] Preferably, the MSQformer model uses the sum of cross entropy classification loss, binary mask loss and edge loss as the total loss, where the mask loss includes weights λ and focal =20.0 and λ dice =1.0 focal loss and dice loss:

[0020] L class =L cross_entropy (c,c gt ) (1)

[0021] L mask =λ focal ·L focal (m,m gt )+λ dice ·L dice (m,m gt ) (2)

[0022] In the formula, c and c gt The predicted category and the true category, m and m gt They are the predicted mask and the real mask respectively;

[0023] By calculating the edge loss of the fused mask, the boundary prediction performance of the mask at each scale is supervised as follows:

[0024] L edge =L focal (e,e gt ) (3)

[0025] In the formula, e and e gt They are the edges extracted from the predicted mask and the real mask respectively.

[0026] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0027] The present invention provides a Maskformer cultivated land parcel identification method based on multi-scale query, the method comprising constructing an MSQformer model based on a Maskformer model, wherein the MSQformer model comprises a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module and a segmentation prediction module, the MSQformer model is trained based on a preset data set to obtain a trained MSQformer model, and cultivated land parcel identification is performed based on the trained MSQformer model. The present invention has the following advantages:

[0028] (1) To address the multi-scale and multi-target problems of Maskformer in farmland extraction tasks, a multi-query design is introduced in Maskformer to increase the accuracy of mask prediction through multiple queries. At the same time, combined with the multi-scale design, each query is matched with a feature map at a specific scale to perform mask predictions of different degrees of precision.

[0029] (2) A fusion module is used to gradually fuse information of different scales to obtain refined mask predictions;

[0030] (3) An edge loss function is designed to supervise the fused mask to ensure that its segmentation boundary is as close as possible to the actual farmland boundary. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0033] Figure 2 Schematic diagram of the MSQformer model structure. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The purpose of the present invention is to provide a Maskformer cultivated land parcel identification method based on multi-scale query, which increases the accuracy of cultivated land parcel mask prediction by means of multiple queries, and predicts the cultivated land parcel mask based on each query and a feature map of the corresponding scale, thereby improving the prediction accuracy and facilitating use.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a Maskformer farmland plot recognition method based on multi-scale query, comprising:

[0038] Step 100: constructing a MSQformer model based on the Maskformer model, wherein the MSQformer model includes a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module and a segmentation prediction module;

[0039] The structural diagram of the MSQformer model is as follows: Figure 2 As shown in the figure, (a) the backbone network is used to extract multi-scale features, and the final feature map is used as the embedded feature input to the transformer decoder for decoding; (b) the prediction mask is generated by the feature map at each scale and the query vector; (c) the multi-scale mask is fused step by step; (d) the fused mask is used with the category embedding vector for prediction;

[0040] Step 200: training the MSQformer model based on a preset data set to obtain a trained MSQformer model;

[0041] Step 300: Identify cultivated land plots based on the trained MSQformer model.

[0042] The multi-scale feature extraction and embedding module is used to extract multi-scale features through the backbone network, and input the final feature map as the embedded feature into the transformer decoder for decoding, specifically:

[0043] Input RGB∈R H×W The image is extracted through the backbone network ResNet50 to obtain a series of reduced resolution feature map scales Among them C F is the channel dimension of the image feature, S i =2 3+i(i=1,2,3,4), the image features with a scale of 1 / 32 have high-level semantic information, which are input into Transformer decoder[] for semantic decoding. Transformer Decoder consists of several stacked Transformer decoder layers. After inputting the embedded features and the query vector, the hidden layer features with the same dimension as the query vector length can be decoded. Since the generation of the cultivated land object mask must have different degrees of refinement at different scales, different numbers of query vectors are generated for the same Transformer Decoder. To generate hidden features of corresponding scale Among them C Q is the query vector dimension, N i is the number of query vectors, C H is the hidden layer feature dimension, i = 1, 2, 3. Although Maskformer uses a feature pyramid to aggregate feature maps of different resolutions to a 1 / 4 scale to obtain image features for predicting masks, it is difficult to predict high-resolution masks using low-resolution features. This ignores the fact that there are different degrees of refinement in the boundaries of cultivated land at different scales. An improvement is made to generate a separate mask for the image feature map at each scale for decision-level fusion, that is, the hidden layer features of these three scales and the three image feature maps of scales 1 / 4, 1 / 8 and 1 / 16 are used for subsequent multi-scale mask generation.

[0044] The mask generation module is used to generate a prediction mask according to the hidden layer features and influencing features at each scale, specifically:

[0045] After obtaining the one-to-one correspondence between hidden layer features and image features at multiple scales, a prediction mask is generated at each scale, such as Figure 2 As shown in part b of , for the binary classification task of farmland extraction, the process divides the image into two regions and uses a binary mask {m i |m i ∈[0,1] H×W}, where i = 0, 1 indicates that the category is background or farmland. Since no instance distinction is performed, all farmland objects are included in a mask as a whole area. The hidden layer features contain the global mask information of each category. The hidden layer features and image features are respectively passed through the fully connected layer and the convolution layer to obtain the mask embedding vector and pixel embedding vector Multiply the two together to get the prediction Where S i =2 3+i And i=1,2,3.

[0046] The multi-scale mask fusion module is used to fuse multi-scale prediction masks, specifically:

[0047] Predicting all farmland plots in only one mask is prone to omission or merging of plots. The farmland plot masks predicted at different scales have different degrees of refinement in their boundaries. Aggregating multiple scales, especially high-resolution scale masks, helps reduce the generation of erroneous boundaries. After generating masks of scales 1 / 4, 1 / 8, and 1 / 16, masks of different scales are fused step by step through a fusion module. Where S i =2 3+i And i = 1, 2, 3, such as Figure 2 As shown in part c of the figure, in this module, the low-resolution mask is fused with the higher-resolution mask through a layer of convolution and upsampling operations, and the fusion of masks of different resolutions at three scales is completed in this way to obtain finer plots and boundaries. Finally, the fused Enter the segmentation header for segmentation.

[0048] The segmentation prediction module is used to perform segmentation prediction according to the fused mask, specifically:

[0049] According to the category prediction method of Maskformer, the hidden layer features of the third scale output by Transformer decoder are used Generate a category prediction vector through a fully connected layer Among them, K is the number of categories, and the prediction result is obtained by performing a dot multiplication operation with fusedmask and using the sigmoid function activation. Finally, the predicted mask is upsampled to the original resolution size to obtain the segmentation result seg∈R K×H×W .

[0050] The MSQformer model uses the sum of cross entropy classification loss, binary mask loss and edge loss as the total loss, where the mask loss includes weights λ and focal =20.0 and λ dice =1.0 focal loss and diceloss:

[0051] L class =L cross_entropy (c,c gt ) (1)

[0052] L mask =λ focal ·L focal (m,m gt )+λ dice ·L dice (m,mgt ) (2)

[0053] In the formula, c and c gt The predicted category and the true category, m and m gt They are the predicted mask and the real mask respectively;

[0054] By calculating the edge loss of the fused mask, the boundary prediction performance of the mask at each scale is supervised as follows:

[0055] L edge =L focal (e,e gt ) (3)

[0056] In the formula, e and e gt They are the edges extracted from the predicted mask and the real mask respectively.

[0057] The present invention provides an embodiment, and experiments are carried out on a public data set obtained by Jilin-1. The Jilin data set is provided by the 2021 High-Resolution Remote Sensing Image Extraction Farmland Competition, which includes four bands of RGB and NIR, with a spatial resolution of 0.75-1.1m. The RGB band is used in the experiment and uniformly cropped to a size of 640×640. The sizes of the training set and the validation set are 1482 and 1390, respectively.

[0058] ResNet50 pre-trained on the ImageNet dataset was used as the backbone in the experiment. The dimension of the hidden layer features in the Transformerdecoder is 256, the number of attention heads is 4, the dimension of the feedforward layer features is 2048, and the corresponding number of query vectors at the three scales are 100, 50, and 25, respectively. The dimensions of the mask embedding vector and image features in the segmentation head are both 256. The batch sample sizes for training and validation are 4 and 1, respectively. All training processes were completed using the Pytorch framework on an Nvidia GTX 1080Ti graphics card with 11GB of video memory. The training process used the AdamW optimizer, with an initial learning rate of 0.0001, a weight decay of 0.0001, 5000 training times, and the learning rate adjustment method was Poly. The model was validated every 200 training times.

[0059]

[0060] In the formula, base_lr is the initial learning rate, epoch and total_epoch are the current training generation and the total training generation respectively;

[0061] There are four cases, namely TP (True Positive), FP (False Positive), FN (False Negative) and TN (True Negative), according to the difference between the prediction of pixel analogy and its actual situation. According to the above four prediction cases, the corresponding semantic segmentation evaluation indicators such as pixel classification accuracy (PA) and intersection of union (IoU) can be calculated;

[0062] (1) Pixel classification accuracy PA

[0063] The pixel classification accuracy indicates the ratio of the number of correctly predicted pixels to the total number of pixels, that is:

[0064]

[0065] The pixel classification accuracy rate indicates the classification accuracy rate of pixels that actually belong to the i-th category in the predicted value of the i-th category. The calculation process is expressed as:

[0066]

[0067] Furthermore, the average pixel classification accuracy is the average of the pixel classification accuracy of all categories, that is:

[0068]

[0069] (2) Intersection over Union (IoU)

[0070] The ratio between the intersection and the union of two sets is used to describe the overlap between the two sets. In the semantic segmentation index, it can be expressed as the ratio of the intersection and the union of the two sets of a certain category prediction result and the true label. The ratio is between [0,1], and the closer it is to 1, the higher the overlap. The calculation process is:

[0071]

[0072] Where X and Y represent two sets of the same size, corresponding to the predicted value set and the true value set in the semantic segmentation problem, and |·| represents the counting operation of the number of elements in the set. Similarly, the average intersection-over-union ratio of all categories can be calculated;

[0073]

[0074] Experimental results:

[0075] In order to illustrate the advantages of the proposed method, representative U-Net and Deeplabv3+ models are selected for comparative experiments on the Jilin dataset:

[0076] Table 1 Comparison of cultivated land extraction results of various methods on Jilin dataset

[0077]

[0078] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0079] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A Maskformer farmland plot recognition method based on multi-scale query, characterized in that: include: The MSQformer model is constructed based on the Maskformer model, wherein the MSQformer model includes a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module and a segmentation prediction module; The MSQformer model is trained based on a preset data set to obtain a trained MSQformer model; Cultivated land plots are identified based on the trained MSQformer model.

2. The method according to claim 1, characterized in that: The multi-scale feature extraction and embedding module is used to extract multi-scale features through the backbone network, and input the final feature map as the embedded feature into the transformer decoder for decoding, specifically: The collected image RGB∈R H×W Input the backbone network ResNet50 for feature extraction to obtain the reduced resolution feature map scale Among them, C F is the channel dimension of the image feature, S i =2 3+i (i=1,2,3,4), and input it into Transformer decoder for semantic decoding. Transformer Decoder is composed of several stacked Transformer Decoder layers, and different numbers of query vectors are generated for the same Transformer Decoder. To generate hidden features of corresponding scale Among them C Q is the query vector dimension, N i is the number of query vectors, C H is the hidden layer feature dimension, i = 1, 2, 3, and finally the hidden layer features and image features with one-to-one correspondence at multiple scales are obtained.

3. The method according to claim 2, characterized in that The mask generation module is used to generate a prediction mask according to the hidden layer features and influencing features at each scale, specifically: The image is divided into two regions and a binary mask {m i |m i ∈[0,1] H×W }, where i = 0, 1 indicates that the category is background or farmland. All farmland objects are included in a mask as a whole area. The hidden layer features contain the global mask information of each category. The hidden layer features and image features are passed through the fully connected layer and the convolution layer respectively to obtain and Multiply the two together to get the prediction Among them, S i =2 3+i And i=1,2,3.

4. The method according to claim 3, characterized in that The multi-scale mask fusion module is used to fuse multi-scale prediction masks, specifically: Based on the multi-scale mask fusion module, different scales are gradually fused Where S i =2 3+i And i = 1, 2, 3, where the low-resolution mask is fused with the higher-resolution mask through a layer of convolution and upsampling operations, and so on to complete the fusion of masks of different resolutions at three scales to obtain finer plots and boundaries, and finally Input segmentation prediction module for segmentation.

5. The method according to claim 4, characterized in that The segmentation prediction module is used to perform segmentation prediction according to the fused mask, specifically: Hidden features of the third scale based on Transformer decoder output Generate categories through a fully connected layer Among them, K is the number of categories, and the prediction result is obtained by performing a dot multiplication operation with the fused mask and using the sigmoid function activation. The predicted mask is upsampled to the original resolution size to obtain the segmentation result 6. The method according to claim 1, characterized in that The MSQformer model uses the sum of cross entropy classification loss, binary mask loss and edge loss as the total loss, where the mask loss includes weights λ and focal =20.0 and λ dice =1.0 focal loss and dice loss: L class =L cross_entropy (c,c gt ) (1) L mask =λ focal ·L focal (m,m gt )+λ dice ·L dice (m,m gt ) (2) In the formula, c and c gt are the predicted category and the true category, m and m gt They are the predicted mask and the real mask respectively; By calculating the edge loss of the fused mask, the boundary prediction performance of the mask at each scale is supervised as follows: THE edge =L focal (and and gt ) (3) In the formula, e and e gt They are the edges extracted from the predicted mask and the real mask respectively.

Images