A Maskformer Cultivated Land Plot Recognition Method Based on Multi-Scale Query

Abstract

The present invention provides a Maskformer-based cultivated land plot recognition method based on multi-scale queries. The method includes: constructing an MSQformer model based on the Maskformer model, where the MSQformer model includes a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module, and a segmentation prediction module; training the MSQformer model based on a preset data set to obtain a trained MSQformer model; and performing cultivated land plot recognition based on the trained MSQformer model. The present invention increases the accuracy of cultivated land block mask prediction by means of multiple queries, and at the same time predicts the cultivated land block mask based on each query and the feature map of the corresponding scale, improving the prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of cultivated land plot recognition, and in particular to a Maskformer cultivated land plot recognition method based on multi-scale query. Background Art

[0002] As the basic unit of farmland, cultivated land plots are the key to farmland information extraction, which is very important for agricultural information investigation, agricultural development and yield estimation, and planting pattern promotion. With the increasing availability of high-resolution remote sensing images, research on the automatic extraction of cultivated land boundaries has received favorable support. However, due to the influence of different imaging conditions, complex background features, and changing time phases, it remains a challenge to intelligently obtain high-precision farmland boundaries based on high-resolution remote sensing images.

[0003] Deep neural networks have made significant breakthroughs in many computer vision tasks. Semantic segmentation networks such as U-net and Deeplab can provide pixel-by-pixel predictions and are applied to the cultivated land extraction task. Most deep learning-based cultivated land extraction works combine semantic segmentation with edge detection to improve the detection accuracy of plot boundaries. Existing methods are usually based on convolutional neural networks, which can automatically implicitly learn high-level depth features from training data through local connection, weight sharing, and sampling. However, such methods often lack the ability to capture long-range dependencies, and their extraction performance is limited in more extensive applications. Since the application of ViT in the field of computer vision, Transformer-based models have achieved remarkable achievements with their superior long-range modeling capabilities and have become the new standard paradigm for semantic segmentation. On this basis, some people have improved and applied the Transformer method to the cultivated land extraction task, providing a research reference for improving the cultivated land extraction accuracy in complex scenarios. Considering that the cultivated land plot object is a closed and continuous area, the method based on mask classification may be a new idea. One of the representative models for mask classification based on Transformer is Maskformer, which is particularly suitable for processing image features with large intra-class variance and small inter-class variance in remote sensing scenarios. Mask classification is to predict a set of binary masks and assign a single category to each mask.

[0004] However, applying Maskformer to cultivated land extraction still faces some problems. First, Maskformer generates masks through downsampled feature maps. In cultivated land extraction, there are often multi-scale effects and a large number of cultivated land areas, resulting in the complex contours of cultivated land blocks being ignored or becoming ambiguous. Second, Maskformer performs semantic segmentation on cultivated land tasks with a single category but a large number of objects. In essence, it combines all cultivated land blocks into one object to predict the mask, and it is difficult to find and accurately identify all cultivated land blocks through a single query vector, which inevitably causes some land blocks to be missed and land blocks to be merged. Therefore, it is very necessary to design a Maskformer cultivated land block recognition method based on multi-scale queries. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, the purpose of the present invention is to provide a Maskformer cultivated land block recognition method based on multi-scale queries.

[0006] To achieve the above purpose, the present invention provides the following solutions:

[0007] The present invention provides a Maskformer cultivated land block recognition method based on multi-scale queries, including:

[0008] Construct an MSQformer model based on the Maskformer model. The MSQformer model includes a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module, and a segmentation prediction module;

[0009] Train the MSQformer model based on a preset data set to obtain a trained MSQformer model;

[0010] Perform cultivated land block recognition based on the trained MSQformer model.

[0011] Preferably, the multi-scale feature extraction and embedding module is used to extract multi-scale features through a backbone network and input the final feature map as embedded features into the transformer decoder for decoding. Specifically:

[0012] Input the collected image RGB ∈ R H×W into the backbone network ResNet50 for feature extraction to obtain the downsampled feature map scale where C F is the channel dimension of the image features, and S i = 2 3+i(i = 1, 2, 3, 4), input it into the Transformer decoder for semantic decoding. Among them, the Transformer Decoder is composed of several stacked Transformer decoder layers, generating different numbers of query vectors for the same Transformer Decoder to generate hidden layer features corresponding to the corresponding scale where C Q is the dimension of the query vector, N i is the number of query vectors, C H is the dimension of the hidden layer feature, i = 1, 2, 3, and finally, the corresponding hidden layer features and image features at multiple scales are obtained.

[0013] Preferably, the mask generation module is used to generate a prediction mask according to the hidden layer features and influence features at each scale, specifically:

[0014] The image is divided into 2 regions, represented by a binary mask {m i |m i ∈[0, 1] H×W}, where i = 0, 1 represents the category of background or cultivated land. All cultivated land objects are included as a whole region in one mask. The hidden layer features contain the global mask information of each category. The hidden layer features and image features are respectively passed through a fully connected layer and a convolutional layer to obtain a mask embedding vector and a pixel embedding vector Multiply the two to get the prediction where, S i = 2 3+i and i = 1, 2, 3.

[0015] Preferably, the multi-scale mask fusion module is used to fuse the prediction masks at multiple scales, specifically:

[0016] Based on the multi-scale mask fusion module, fuse the at different scales step by step. Among them, S i = 2 3+i and i = 1, 2, 3. Among them, the low-resolution mask is fused with the higher-resolution mask through a layer of convolution and upsampling operations, and so on to complete the fusion of different-resolution masks at three scales, used to obtain finer plots and boundaries. Finally, is input into the segmentation prediction module for segmentation.

[0017] Preferably, the segmentation prediction module is used to perform segmentation prediction according to the fused mask, specifically:

[0018] The hidden layer features of the third scale output by the Transformer decoder Generate a class prediction vector through a fully connected layer where K is the number of classes, perform a dot product operation with the fused mask and activate it using the sigmoid function to obtain the prediction result, and upsample the prediction mask to the original resolution size to obtain the segmentation result seg∈R K×H×W .

[0019] Preferably, the MSQformer model uses the sum of cross-entropy classification loss, binary mask loss, and edge loss as the total loss, where the mask loss includes focal loss and dice loss with weights λ focal = 20.0 and λ dice = 1.0 respectively:

[0020] L class = L cross_entropy (c, c gt ) (1)

[0021] L mask = λ focal ·L focal (m, m gt ) + λ dice ·L dice (m, m gt ) (2)

[0022] In the formula, c and c gt are the predicted class and the true class respectively, and m and m gt are the predicted mask and the true mask respectively;

[0023] By calculating the edge loss for the fused mask, the boundary prediction performance of the mask at each scale is supervised, which is:

[0024] L edge = L focal (e, e gt ) (3)

[0025] In the formula, e and e gt are the edges extracted from the predicted mask and the true mask respectively.

[0026] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0027] The present invention provides a Maskformer-based cultivated land plot recognition method based on multi-scale query. The method includes constructing an MSQformer model based on the Maskformer model. The MSQformer model includes a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module, and a segmentation prediction module. The MSQformer model is trained based on a preset data set to obtain a trained MSQformer model, and cultivated land plots are recognized based on the trained MSQformer model. The present invention has the following advantages:

[0028] (1) Aiming at the multi-scale and multi-object problems existing in the cultivated land extraction task of Maskformer, a multi-query design is introduced into Maskformer. By querying multiple times, the accuracy of mask prediction is increased. At the same time, combined with the multi-scale design, each query is corresponded to the feature map at a specific scale for mask prediction with different levels of fineness.

[0029] (2) A fusion module is used to gradually fuse information at different scales to obtain refined mask prediction.

[0030] (3) An edge loss function is designed to supervise the fused mask, ensuring that its segmentation boundary is closest to the actual cultivated land boundary to the greatest extent. Description of the Drawings

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1 It is a flowchart of the method provided by the embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of the MSQformer model structure. Detailed Embodiments

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0035] The objective of the present invention is to provide a Maskformer cultivated land plot recognition method based on multi-scale queries, which increases the accuracy of cultivated land plot mask prediction by means of multiple queries. Meanwhile, the cultivated land plot mask is predicted based on each query and the feature map of the corresponding scale, improving the prediction accuracy and facilitating use.

[0036] To make the above objectives, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Figure 1 The flowchart of the method provided by the embodiment of the present invention is as Figure 1 As shown, the present invention provides a Maskformer cultivated land plot recognition method based on multi-scale queries, including:

[0038] Step 100: Construct an MSQformer model based on the Maskformer model, where the MSQformer model includes a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module, and a segmentation prediction module;

[0039] The structural schematic diagram of the MSQformer model is as Figure 2 As shown, where (a) the backbone network is used to extract multi-scale features, and the final feature map is used as the embedded feature to input into the transformer decoder for decoding; (b) the predicted mask is generated from the feature map and query vector at each scale; (c) the multi-scale masks are fused step by step; (d) the fused mask and the category embedding vector are used for prediction;

[0040] Step 200: Train the MSQformer model based on a preset dataset to obtain a trained MSQformer model;

[0041] Step 300: Identify cultivated land plots based on the trained MSQformer model.

[0042] The multi-scale feature extraction and embedding module is used to extract multi-scale features through the backbone network and input the final feature map as the embedded feature into the transformer decoder for decoding. Specifically:

[0043] The input RGB ∈ R H×W The image is subjected to feature extraction by the backbone network ResNet50 to obtain a series of downsampled feature map scales where C F is the channel dimension of the image features, and S i = 2 3+i(i = 1, 2, 3, 4), the image features at a scale of 1 / 32 have high-level semantic information, which are input into the Transformer decoder [] for semantic decoding. The Transformer Decoder is composed of several stacked Transformer decoder layers. After inputting the embedded features and query vectors, the hidden layer features with the same dimension as the length of the query vector can be decoded. Since the generation of the cultivated land object mask must have different levels of fineness at different scales, different numbers of query vectors are generated for the same Transformer Decoder to generate the corresponding hidden layer features at the scale where C Q is the dimension of the query vector, N i is the number of query vectors, C H is the dimension of the hidden layer features. i = 1, 2, 3. Although Maskformer uses a feature pyramid to aggregate feature maps with different resolutions to the 1 / 4 scale to obtain the image features for predicting the mask, it is difficult to predict the high-resolution mask using low-resolution features, which ignores the characteristics of different levels of fineness of the cultivated land block boundaries at different scales. To improve this, separate masks are generated for the image feature maps at each scale for decision-level fusion, that is, the hidden layer features at these three scales and the three image feature maps at scales of 1 / 4, 1 / 8, and 1 / 16 are used for the subsequent generation of multi-scale masks.

[0044] The mask generation module is used to generate the predicted mask according to the hidden layer features and influence features at each scale. Specifically:

[0045] After obtaining the corresponding hidden layer features and image features at multiple scales, the predicted mask is generated at each scale. As shown in part b of Figure 2 , for the binary classification task of cultivated land extraction, this process divides the image into 2 regions, represented by the binary mask {m i |m i ∈[0, 1] H×W}, where i = 0, 1 represents the category of background or cultivated land. Since no instance distinction is made, all cultivated land objects are included as a whole region in one mask. The hidden layer features contain the global mask information of each category. The hidden layer features and image features are respectively passed through the fully connected layer and the convolutional layer to obtain the mask embedding vector and the pixel embedding vector The two are multiplied to obtain the prediction where S i = 2 3+i and i = 1, 2, 3.

[0046] The multi-scale mask fusion module is used to fuse the predicted masks at multiple scales, specifically as follows:

[0047] Predicting all arable land blocks in only one mask is very likely to result in omissions or block mergers. The arable land block masks predicted at different scales have different levels of refinement at the boundaries. Aggregating masks at multiple scales, especially high-resolution scales, helps reduce the generation of incorrect boundaries. After generating masks at scales of 1 / 4, 1 / 8, and 1 / 16, through a fusion module, the masks at different scales are fused step by step. where S i = 2 3+i and i = 1, 2, 3, as shown in part c of Figure 2 In this module, the low-resolution mask is fused with the higher-resolution mask through a layer of convolution and upsampling operations. And so on, the fusion of masks with different resolutions at three scales is completed to obtain more refined plots and boundaries. Finally, the fused is input into the segmentation head for segmentation.

[0048] The segmentation prediction module is used to perform segmentation prediction based on the fused mask, specifically as follows:

[0049] According to the category prediction method of Maskformer, the hidden layer features of the third scale output by the Transformer decoder are used to generate the category prediction vector through a layer of fully connected layers where K is the number of categories, which is dot-multiplied with the fused mask and activated using the sigmoid function to obtain the prediction result. Finally, the predicted mask is upsampled to the original resolution size to obtain the segmentation result seg ∈ R K×H×W .

[0050] The MSQformer model uses the sum of the cross-entropy classification loss, binary mask loss, and edge loss as the total loss. Among them, the mask loss includes the focal loss and dice loss with weights of λ focal = 20.0 and λ dice = 1.0 respectively:

[0051] L class = L cross_entropy (c, c gt ) (1)

[0052] L mask = λ focal · L focal (m, m gt ) + λ dice · L dice (m, mgt ) (2)

[0053] Where c and c gt are the predicted category and the true category respectively, and m and m gt are the predicted mask and the true mask respectively;

[0054] By calculating the edge loss of the fused mask, the boundary prediction performance of the mask at each scale is supervised, which is:

[0055] L edge = L focal (e, e gt ) (3)

[0056] Where e and e gt are the edges obtained by extracting the predicted mask and the true mask respectively.

[0057] The present invention provides an embodiment, and experiments are carried out on the public dataset obtained by Jilin-1. The Jilin dataset is provided by the competition of extracting cultivated land from high-resolution remote sensing images in 2021, including four bands of RGB and NIR, with a spatial resolution of 0.75 - 1.1m. In the experiment, the RGB band is used and uniformly cropped to a size of 640×640. The sizes of the training set and the validation set are 1482 and 1390 respectively.

[0058] In the experiment, ResNet50 pre-trained on the ImageNet dataset is used as the backbone. The dimension of the hidden layer features in the Transformer decoder is 256, the number of attention heads is 4, the dimension of the feed-forward layer features is 2048, and the number of query vectors corresponding to the three scales are 100, 50, and 25 respectively. The dimensions of the mask embedding vector and the image features in the segmentation head are both 256. The batch sample sizes for training and validation are 4 and 1 respectively. All training processes are completed using the Pytorch framework on an Nvidia GTX ********* with 11GB of video memory. The AdamW optimizer is used in the training process, the initial learning rate is 0.0001, the weight decay is 0.0001, the number of training times is 5000, the learning rate adjustment method is Poly, and the model is verified every 200 training times;

[0059]

[0060] Where base_lr is the initial learning rate, epoch and total_epoch are the current training generation and the total training generation respectively;

[0061] According to the differences between the predicted situation and the actual situation based on pixel analogy, there are four situations in total, namely TP (True Positive), FP (False Positive), FN (False Negative), and TN (True Negative). Based on the above four prediction situations, corresponding semantic segmentation evaluation metrics can be calculated, such as Pixel Accuracy (PA) and Intersection of Union (IoU).

[0062] (1) Pixel Accuracy PA

[0063] Pixel Accuracy represents the proportion of the number of pixels predicted correctly to the total number of pixels, that is:

[0064]

[0065] And Pixel Classification Class Accuracy represents the Pixel Accuracy of the pixels that truly belong to the i-th category among the predicted values of the i-th category. The calculation process is expressed as:

[0066]

[0067] Furthermore, Pixel Classification Average Accuracy is the average of the Pixel Classification Class Accuracies of all categories, that is:

[0068]

[0069] (2) Intersection of Union IoU

[0070] The ratio between the intersection and the union of two sets is used to describe the overlap degree between the two sets. In semantic segmentation metrics, it can be expressed as the ratio of the intersection and the union of the predicted result of a certain category and the true label. The ratio is between [0,1], and the closer it is to 1, the higher the overlap degree. The calculation process is as follows:

[0071]

[0072] Where X and Y respectively represent two sets of the same size, corresponding to the predicted value set and the true value set in the semantic segmentation problem, and |·| represents the operation of counting the number of elements in the set. Similarly, the average Intersection of Union of all categories can be calculated;

[0073]

[0074] Experimental Results:

[0075] To illustrate the advantages of the proposed method, representative U-Net and Deeplabv3+ models are selected for comparative experiments on the Jilin dataset:

[0076] Table 1 Comparison table of cultivated land extraction results of various methods on Jilin dataset

[0077]

[0078] In this specification, each embodiment is described in a progressive manner. What each embodiment focuses on is the difference from other embodiments. For the same and similar parts among the embodiments, reference can be made to each other.

[0079] In this article, specific examples are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A Maskformer farmland parcel identification method based on multi-scale query, characterized in that: include: The MSQformer model is constructed based on the Maskformer model. The MSQformer model includes a multi-scale feature extraction and embedding module, a mask generation module, a multi-scale mask fusion module, and a segmentation prediction module. The MSQformer model is trained based on a preset data set to obtain a trained MSQformer model; Identify cultivated land plots based on the trained MSQformer model; The multi-scale feature extraction and embedding module is used to extract multi-scale features through the backbone network and input the final feature map as the embedded feature into the transformer decoder for decoding. Specifically: The collected image RGB∈R H×W Input the backbone network ResNet50 for feature extraction to obtain the reduced resolution feature map scale Among them, C F is the channel dimension of the image feature, S i =2 3+i (i=1,2,3,4), and input it into Transformer decoder for semantic decoding. Transformer Decoder is composed of several stacked Transformer Decoder layers, and different numbers of query vectors are generated for the same Transformer Decoder. To generate hidden features of corresponding scale Among them C Q is the query vector dimension, N i is the number of query vectors, C H is the hidden layer feature dimension, i = 1, 2, 3, and finally the hidden layer features and image features with one-to-one correspondence at multiple scales are obtained.

2. The method according to claim 1, characterized in that The mask generation module is used to generate a prediction mask based on the hidden layer features and image features at each scale, specifically: The image is divided into two regions and a binary mask {m i |m i ∈[0,1] H×W }, where i = 0, 1 indicates that the category is background or farmland. All farmland objects are included in a mask as a whole area. The hidden layer features contain the global mask information of each category. The hidden layer features and image features are passed through the fully connected layer and the convolution layer respectively to obtain the mask embedding vector and pixel embedding vector Multiply the two to get the prediction Among them, S i =2 3+i And i=1,2,3.

3. The method according to claim 2, characterized in that The multi-scale mask fusion module is used to fuse multi-scale prediction masks, specifically: Based on the multi-scale mask fusion module, different scales are gradually integrated Among them S i =2 3+i And i = 1, 2, 3, where the low-resolution mask is fused with the higher-resolution mask through a layer of convolution and upsampling operations, and so on to complete the fusion of masks of different resolutions at three scales to obtain finer plots and boundaries, and finally the fused Input segmentation prediction module for segmentation.

4. The method according to claim 3, characterized in that The segmentation prediction module is used to perform segmentation prediction based on the fused mask, specifically: The third scale hidden layer features based on the Transformer decoder output Generate a category prediction vector through a fully connected layer Among them, K is the number of categories, and the prediction result is obtained by performing a dot multiplication operation with the fused mask and activating it with the sigmoid function. The predicted mask is upsampled to the original resolution size to obtain the segmentation result seg∈R K×H×W .

5. The method according to claim 1, characterized in that The MSQformer model uses the sum of cross entropy classification loss, binary mask loss and edge loss as the total loss, where the mask loss includes weights λ and focal =20.0 and λ dice =1.0 focal loss and dice loss: L class =L cross_entropy (c,c gt ) (1) L mask =λ focal ·L focal (m,m gt )+λ dice ·L dice (m,m gt ) (2) Where c and c gt The predicted category and the true category, m and m respectively gt They are the predicted mask and the real mask respectively; By calculating the edge loss of the fusion mask, the boundary prediction performance of the mask at each scale is supervised, which is: THE edge =I focal (and and gt ) (3) Where, e and e gt They are the edges extracted from the predicted mask and the real mask respectively.

Images