Remote sensing image segmentation method based on cross-modal vision-text fusion

Abstract

The invention provides a remote sensing image segmentation method based on cross-modal vision-text fusion, and a subject network comprises an image encoder capable of extracting image features, a language encoder capable of extracting language features, a cross-modal vision language feature fusion device capable of carrying out segmentation guidance, and an image decoder module design. A non-critical filter is designed to enable the model to be more focused on keywords in language description, a central point predictor is designed to enhance the ability of the model to identify a difficult target, a target region predictor is designed to enable the model to find out a region which is most likely to contain the target, and a false alarm predictor is designed to deal with the situation of describing a false target. And finally, building a remote sensing image indication segmentation network model based on cross-modal vision-text fusion, and carrying out network model training and parameter optimization. All modules of the network model are based on a Transform model, an attention mechanism is introduced to capture the relationship dependence between local features and global features of a remote sensing image and the relationship dependence between each region of the remote sensing image and language description, and a cross-modal visual language feature fusion device is used to make the model pay more attention to describe the region where a target is located. And finally, identifying a specifically described target from the remote sensing scene image.

Description

Technical Field

[0001] The present invention relates to a cross-modal image recognition technology, and in particular to a remote sensing image segmentation method based on cross-modal vision-text fusion, which belongs to the cross-technical field of computer vision and natural language processing. Background Art

[0002] Image segmentation is an important research field in computer vision, and there are a lot of studies in the field of remote sensing scene images. Traditional image segmentation can identify all targets in an image, but cannot identify specific targets. How to enable computers to understand human natural language, observe the world from a human perspective based on semantics, and make corresponding operations has become a research hotspot. Indicator segmentation is a basic visual language cross-modal fusion task. Given an image and a natural language expression, the goal is to generate a segmentation mask in the image that corresponds to the entity referred to by the natural language expression. In the field of remote sensing scene images, indicator segmentation has three major difficulties: the difficulty brought by the characteristics of remote sensing images, the difficulty of aligning image and language features, and the difficulty of how the model identifies key semantic information to improve segmentation quality. Therefore, there is always a lack of corresponding research on indicator segmentation of remote sensing images.

[0003] As early as 2016, there were related studies on indication segmentation at home and abroad. Indication segmentation itself has two major difficulties. The first is how to align language features with visual features well, and the second is how to make the model At present, most existing methods follow the rules defined in well-known datasets such as ReferIt and RefCOCO, and have made great progress in recent years. There are two main approaches over the years: the first is that before the Transformer was introduced into computer vision, convolutional neural networks and long short-term memory recurrent neural networks were used to extract visual and language features respectively, and then spliced ​​and fused to obtain the final segmentation mask, but the biggest problem with this method is that it simply splices visual features with language features, and cannot fully capture the global interaction information that is crucial to semantic understanding and segmentation. The second is that after the Transformer was introduced into computer vision, both the visual encoder and the language encoder perform self-attention operations based on the Transformer architecture. The obtained visual features and language features are not simply spliced, but use mutual attention operations, which can fully obtain the interactive relationship between the global and the local.

[0004] However, the above studies are all based on the indication segmentation research of natural scene images. Due to the characteristics of remote sensing scene images such as small targets, dense targets, large target scale variations, and irregular target shapes, there is still no relevant research on remote sensing scene indication segmentation algorithms. If the natural scene indication segmentation algorithm model is directly transferred to the remote sensing scene, the prediction effect needs to be improved.

[0005] The remote sensing scene image indication segmentation algorithm proposed in this patent is aimed at the need to identify specific targets in remote sensing scenes. At the same time, combined with the importance of many current applications of remote sensing scene images for national defense, environmental monitoring, smart cities, etc., a remote sensing image segmentation algorithm based on cross-modal vision-text fusion is designed to achieve the requirement of correctly identifying specific targets in remote sensing scenes. Summary of the invention

[0006] Purpose of the Invention

[0007] The purpose of the present invention is to overcome the difficulties in the above-mentioned remote sensing image indication segmentation and provide an image segmentation method based on cross-modal vision-text fusion. This method models global context information through cross-modal vision-text fusion to identify specific targets described in text in the image, and improves the model according to the characteristics of remote sensing images, thereby effectively improving the segmentation quality in remote sensing scenarios.

[0008] Technical Solution

[0009] The present invention provides a remote sensing image segmentation method based on cross-modal vision-text fusion, and the steps are as follows:

[0010] Step 1: Input image and language description are respectively extracted features through image encoder and language encoder;

[0011] Collect and build a remote sensing scene indication segmentation dataset with a size of 12,000 instances. After the dataset is built, it is planned to use the existing Transformer-based image pre-training model and language pre-training model as the image feature extractor and language feature extractor in the network model. The image feature extractor uses the Swin Transformer pre-training model, and the language feature extractor uses the BERT pre-training model.

[0012] Step 2: The visual language fusion module is used to fuse image features with language features, with the goal of adjusting the attention paid to each area of ​​the image through language features;

[0013] First, the input image is divided into P*P regions, and the size of each region is dynamically adaptive. Then, each region is encoded, and position encoding is added, and self-attention operation is performed to encode the relationship between the regions of the image, and finally the region that may contain the target can be predicted. Secondly, the features obtained from the above only contain the relationship between the regions of the image, so the mutual attention operation is performed on this image feature and the language feature of the language feature extractor to detect the relationship between the language feature vectors and the regions of the image, that is, to find the region containing the language description target on the image according to the language input.

[0014] Step 3: Design a target region predictor to let the model find the region most likely to contain the target.

[0015] The output feature vector from the visual language fusion module obtained in step 2 is directly subjected to a multi-layer perceptron operation to obtain P×P features, which are then concatenated to obtain a feature map of the same size as the original image. The activation value of the area that actually contains the target is larger, which is used to predict whether each area in the image contains the target.

[0016] Step 4: Design a false alarm predictor module based on the false target description, and decide whether to output a segmentation mask or a false alarm signal under the control of the false alarm predictor.

[0017] The output feature vector from the visual language fusion module obtained in step 2 is subjected to a global pooling operation. Each feature vector represents whether a region contains a described target. Finally, a multi-layer perceptron is used to predict whether it contains a target, i.e., 0 / 1 prediction. If the output is binarized by a threshold and the value is 1, the final output is the segmentation mask obtained by superimposing the segmentation mask output by the output graphic decoder and the feature map output by the target region predictor; if the output is binarized by a threshold and the value is 0, the final output is a segmentation mask without a target.

[0018] Step 5: The image features output by the image encoder are decoded by the image decoder to obtain a feature map of the same size as the original image, and the final segmentation mask is output under the guidance of the visual language fusion module.

[0019] The image decoder obtains multi-scale image features from the image encoder, passes the multi-scale image features through the deformable Transformer to obtain multi-scale features with the same dimension as the input, and then upsamples the high-dimensional features and fuses them with the low-dimensional features to finally obtain a segmentation mask with the same size as the original image.

[0020] Step 6: Based on the difficulties brought by the characteristics of remote sensing images, non-keyword filters and center point predictors are designed to make the network model pay more attention to keywords in language descriptions and identify difficult targets.

[0021] The non-keyword filter has a simple structure. It mainly uses the natural language corpus to extract non-keywords in sentences that contain no or very little semantic information, and also sends the filtered text to the language encoder to obtain its features. The center point predictor receives the language features of the original sentence and the filtered sentence and the image features output by the image encoder, and performs mutual attention operations on them respectively, and finally performs fusion decoding to obtain a Gaussian circle prediction map with the center at the target center point. The supervised center point prediction branch formed by these two consecutive operations allows the model to better understand the keywords in the input sentence and improve the model's ability to identify difficult targets through center point prediction. At the same time, the original sentence features and keyword features are obtained. The model not only pays attention to all language information, but also pays better attention to key language information, so as to better identify the corresponding target on the image.

[0022] Through the above steps, with the help of remote sensing scene images and language descriptions, the remote sensing image segmentation model with cross-modal vision-text fusion can identify specific targets described in the image. The visual language fusion module is used to fuse the obtained image features and language features, the false alarm predictor determines the output form of the segmentation head, and the model recognition ability is improved through non-keyword filtering and center point prediction, finally realizing language-guided remote sensing scene specific target recognition.

[0023] Advantages of the invention

[0024] The advantages of the present invention are that good specific target recognition is achieved on the established remote sensing scene indication segmentation data set, and the model is improved according to the remote sensing scene image characteristics, thereby improving the effect of the model in predicting remote sensing scene targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Network framework of remote sensing image segmentation model based on cross-modal vision-text fusion.

[0026] Figure 2 Attention mechanism. DETAILED DESCRIPTION

[0027] The present invention is a remote sensing image segmentation method based on cross-modal vision-text fusion. Figure 1 and Figure 2 As shown, the steps are as follows:

[0028] S1, input image and language description are extracted through image encoder and language encoder respectively, the content is as follows:

[0029] S1.1. Construct a remote sensing scene indication segmentation dataset. Since constructing an indication segmentation dataset requires both segmentation annotation and specific target language description, coupled with the difficulty of annotation due to the characteristics of remote sensing images, there is no large-scale remote sensing indication segmentation dataset. First, the source image is obtained from the classic remote sensing dataset DOTA dataset, and then cropped. The cropped remote sensing scene images are all 1024*1024 size images, and specific targets are found for each image for segmentation annotation and language description. Finally, 12004 instances are obtained, with a total of 15 categories, such as "ship", "car", "bridge", "tennis court" and other common targets in remote sensing scenes. The language description contains the attributes of specific targets such as "red", "big", etc., as well as positional relationships such as "on the right side of...", "in the lower right corner of the picture", etc. Each instance includes ternary information: original image, specific target segmentation annotation, and specific target language description.

[0030] S1.2. Use the Transformer architecture based on the attention mechanism as the basic unit of this network model. The attention mechanism is mainly used to explore the relationship between different regions of the image or different parts of the language. The implementation process is as follows: Figure 2 As shown:

[0031] Here, q can be considered as the input query information, k is the target information corresponding to the query information, and v represents the matching degree between q and k. The derivation process is to first divide the image into n regions or divide the text into n parts by word ( Figure 2 In this case, n=3), and these n parts are reduced in dimension to obtain vectors of fixed length f1, f2, ..., f n , respectively, these n vectors and the learnable matrix W q , W k , W v Get q1, q2, ..., q of each part n with k1, k2, ..., k n and v1, v2, ..., v n For the first vector, calculate q1 and k1, k2, ..., k respectively. n α′ obtained by dot product 1，1 , α′ 1，2 , …, α′ 1，n , and finally output b1. The relevant formula is as follows (taking n=3 as an example):

[0032] Q=[q1 q2 q3]=W q [f1 f2 f3] (1)

[0033] K=[k1 k2 k3]=W k [f1 f2 f3] (2)

[0034] V=[v1 v2 v3]=W v [f1 f2 f3] (3)

[0035]

[0036] The above is an attention operation, and the multi-head self-attention operation aims to allow the model to learn different dependencies, so different W can be used q , W k , W v The original vector is projected into different subspaces, and then self-attention operation is performed. Finally, the results of different subspaces can be spliced ​​as the final output.

[0037]

[0038] A Transformer basic unit architecture not only includes a simple multi-head attention operation, but also needs to obtain the position information of each part through position encoding. That is, f1, f2, ..., f n The vectors obtained by their respective position encodings are added separately, and then the above-mentioned multi-head attention operation is performed. Then, the output is added to the input using a residual connection, and the layer normalization operation is performed, as shown in Formula 6. The final output is obtained by passing through two fully connected layers and performing residual connections and layer normalization again.

[0039] S1.3. For the input remote sensing scene image and specific target language description, the remote sensing scene image uses a Transformer-based image encoder for feature extraction. Here, the image pre-training model is directly used as the image encoder, while the language description uses a Transformer-based natural language pre-training model for feature extraction. The pre-training model can improve the generalization ability of the network, and after freezing the parameters, it can reduce the computational overhead and thus reduce the training time. The input of the image encoder is a remote sensing scene image scaled from 1024*1024 to 480*480, and the output is a feature map of three scales. The purpose is to use the feature pyramid structure to simultaneously obtain deep global semantic information and shallow local specific information. The language description first removes punctuation marks and changes all to lowercase, and then sends it to BERT to obtain the final semantic feature vector. The length of the feature vector here is fixed to 20, and the part exceeding the sentence length is padded with zeros.

[0040] S2. Visual language fusion based on mutual attention mechanism is used to fuse image features and language features. The content is as follows:

[0041] S2.1. First, perform dynamic query operations on each region of the input image. The first step is to convert the image feature F output by the image encoder i∈H×W×C is adaptively and dynamically divided into P×P (P=16) regions, and P learnable regions are introduced to query Q1, Q2, ..., Q P , each region query corresponds to a spatial region in the image, totaling P 2 , responsible for feature decoding of each region. The formula is as follows:

[0042] O ri =softmax(Q r (ρ(F i W ik )) T ) (7)

[0043] F a =O ri (ρ(F i W iv )) T (8)

[0044]

[0045] In formula 7, W ik is a learnable matrix of size C×C, where C is F i The number of channels, ρ is the GELU activation function, and the obtained O ri ∈P 2 ×H×W generates an H×W attention map for each region query. In Formula 8, W iv It is also a learnable matrix of size C×C. In this way, each region in the image can dynamically collect the features of each region and solve the complex feature relationships in the image.

[0046] S2.2, then, get F a After that, it can be further used to model the interaction between regions and regions, and between regions and languages. a The language feature F output by the language encoder l As input, attention operation is performed. The parallel operation is divided into two steps. The first step is to use F a Perform self-attention operation on the input to get F r1 ∈P 2 ×HW. The second step is to use F a As q, the language feature F l As k and v, they perform mutual attention operations to obtain the interdependence of visual language feature fusion, that is, the relationship between each region and each word is modeled, and the output F is obtained. r2 ∈P 2 ×HW. Finally, by element-by-element a 、F r1 、F r2Add and pass through the multi-layer perceptron to get the final output F r The formula is as follows:

[0047] A m =softmax((ρ(F a W mq )) T (ρ(F l W mk )) T ) (10)

[0048] F r =MLP(F a +F r1 +F r2 ) (11)

[0049] Among them, Formula 10 is the visual language mutual attention operation, A m ∈P 2 ×N t , N t is the length of the language feature vector. mq ∈P 2 ×C and W mk ∈N t ×C are all learnable matrices. MLP is a multi-layer perceptron.

[0050] S3. Use a supervised target region predictor to predict regions that may contain targets. The content is as follows:

[0051] S3.1. The input of the target region predictor is the region feature F output by the visual language fusion module. r (P 2 ×C,P 2 is the number of regions the image is divided into, C is the length of the feature vector of each region), f1 r 、f r 2 ,……, F r P with length C 2 feature vectors, perform multi-layer perceptron operations on each of them, and finally generate P 2 ×1 feature maps similar to heat maps, corresponding to P on the original image 2 The mask map actually used for supervision is the mask map of the original segmentation mask map downsampled to a size of P × P. If a region actually contains a described object, the larger the activation value.

[0052]

[0053] As shown in Exhibit 12, MLP is a multi-layer perceptron operation, and Concat is the output of P2 The values ​​are concatenated in sequence to form a mask of size P × P. This branch is used to improve the model's ability to predict whether each area of ​​the image contains an object.

[0054] S4. Design a supervised false alarm predictor for false target descriptions to determine whether the described target is a false target, as follows:

[0055] S4.1. Regional features F output by the visual language fusion module r (P 2 ×C,P 2 is the number of regions the image is divided into, C is the length of the feature vector of each region), firstly, global average pooling is performed on it, that is, the feature vector of each region with a length of C is averagely pooled to obtain P 2 The result of global average pooling is passed through a multi-layer perceptron to obtain a predicted probability between 0 and 1, which is then binarized (0 / 1). If it is 1, it means it is a false target, otherwise the image contains the described target.

[0056]

[0057] As shown in Formula 13, MLP is a multi-layer perceptron, GAP is a global average pooling, and f1 r 、f r 2 , …, F r P with length C 2 feature vector, NT is the predicted probability of whether the image contains the described target.

[0058] This branch is used to supervise whether a specific target described in the language can correspond to the image. When the prediction result is 1, regardless of the segmentation mask obtained by adding the image decoder output and the target area predictor output element by element, a false alarm signal is output, that is, a segmentation mask without any predicted target; when the prediction result is 0, the segmentation mask obtained by adding the image decoder output and the target area predictor output element by element is output. This branch can directly enable the model to have the ability to predict false description targets.

[0059] S5. Use the image decoder to decode the image features and output the segmentation mask of the original image size, and fuse it with the output of the visual language fusion module to obtain the final segmentation mask output of the original image size. The content is as follows:

[0060] S5.1. The image decoder uses the image decoder in the deformable DETR. Deformable convolution DCN is a good mechanism for paying attention to sparse spatial positions, but it lacks the ability to model the relationship between positions. Transformer is very powerful for relationship modeling, so deformable DETR was born. The deformable attention module of the image encoder combines the sparse sampling capability of DCN and the global relationship modeling capability of Transformer. This module can aggregate multi-scale features, so this module replaces the cross-in-interaction attention module in the image decoder. It mainly takes the multi-scale features output by the image encoder as input and outputs the mask feature F m ∈H×W×C.

[0061] S5.2. At the same time, the regional features in S2.1 are output to F a After the multi-layer perceptron, the regional filter F containing regional clues is obtained f ∈P 2 ×C, multiply the matrix to get the region mask feature M r ∈H×W×P 2 , and the P obtained in S3.1 2 ×1 size target area prediction map, and fuse it with it to obtain the final mask output. The specific operation is to 2 ×1 is P 2 values ​​as weights, and M r P 2 The feature maps of each channel are superimposed on a single channel according to the weights, which is the final mask output, and this output is used for supervision and final prediction.

[0062] S6. Use non-keyword filtering and predict specific target center points to improve the model prediction ability. The content is as follows:

[0063] S6.1. The design of the non-keyword filter has a very simple structure. It directly uses the existing corpus to identify non-keywords in the input language description and delete them. That is, "the red car on the road in the upper left corner of the picture" can be obtained after non-keyword filtering as "the red car on the road in the upper left corner of the picture". The filtered sentence is also passed through the language pre-training model to obtain a language feature vector. In this way, the lost semantic information can be ignored, and the model can focus on words containing key information.

[0064] S6.2. The feature vector of the original sentence and the feature vector of the filtered sentence are fused with the image feature vector at the same time and used to predict the center point of a specific object. The center point annotation of the real target is mainly to find the center point in its segmented connected domain based on its original segmentation annotation, and draw a Gaussian circle with the center point as the center. This is for continuous prediction. The closer the predicted center point is to the actual center point, the lower the loss. The center of the circle is the highest probability value 1. The radius of the Gaussian circle is set to one-fifth of the longest side of the smallest rectangle containing the actual segmentation annotation connected domain. This can improve the model's ability to analyze the semantics of the input text and identify difficult targets.

Claims

1. A remote sensing image segmentation method based on cross-modal vision-text fusion, characterized in that: include: Step 1: Collect and build a remote sensing scene indication segmentation dataset. Use an image encoder to extract image features from the remote sensing scene image, and use a language encoder to extract language features from the language description; Step 2: Send the obtained image features and language features to the visual language fusion module for feature fusion, so that the model can adjust the attention of each area of ​​the image through the language features; Step 3: Design a target region predictor to let the model find the region most likely to contain the target. Step 4: Design a false alarm predictor module for false target description. After the segmentation mask output by the graphic decoder is superimposed with the feature map output by the target area predictor, it is decided whether to output it under the control of the false alarm predictor. Step 5: The image features output by the image encoder are decoded by the image decoder to obtain a feature map with the same size as the original image, and the segmentation mask is output under the guidance of the visual language fusion module. Step 6: Based on the characteristics of remote sensing images, design non-keyword filters and center point predictors to make the network model pay more attention to keywords in language descriptions and identify difficult targets. Through the above steps, with the help of remote sensing scene images and language descriptions, the remote sensing image segmentation model of cross-modal vision-text fusion can identify specific targets described in the image. The visual language fusion module is used to fuse the obtained image features and language features, and the false alarm predictor determines the output form of the segmentation head, ultimately realizing the recognition of specific targets in remote sensing scenes.

2. The remote sensing image segmentation method based on cross-modal vision-text fusion according to claim 1, characterized in that: The method of using image encoder and language encoder to extract image features and language features respectively described in "Step 1" is as follows: First, build a remote sensing scene indication segmentation dataset with a size of 12,000 instances. After the dataset is built, it is planned to use the existing Transformer-based image pre-training model and language pre-training model as the image feature extractor and language feature extractor in the network model. Among them, the language input part needs to pass through a non-keyword filter to obtain the keywords in the language input, and then pass through the language pre-training model together with the original sentence to obtain the original sentence features and keyword features, which can better allow the model to perform.

3. The remote sensing image segmentation method based on cross-modal vision-text fusion according to claim 1, characterized in that: The design of the visual language fusion module described in "Step 2" is as follows: First, the input image is divided into P×P regions, and the size of each region is dynamically adaptive. Then, each region is encoded, and position encoding is added, and a self-attention operation is performed to encode the relationship between the regions of the image. Finally, the region that may contain the target can be predicted. Secondly, the features obtained from the above only contain the relationship between the various image regions. Therefore, a mutual attention operation is performed on this image feature and the language feature of the language feature extractor to detect the relationship between the various language feature vectors and the various image regions, that is, to find the area on the image that contains the language description target based on the language input.

4. The remote sensing image segmentation method based on cross-modal vision-text fusion according to claim 1, characterized in that: The target region predictor design described in "Step 3" is done as follows: The output feature vector from the visual language fusion module obtained in step 2 is directly subjected to a multi-layer perceptron operation to obtain P×P features, which are then concatenated to obtain a feature map of the same size as the original image. The activation value of the area that actually contains the target is larger, which is used to predict whether each area in the image contains the target.

5. The remote sensing image segmentation method based on cross-modal vision-text fusion according to claim 1, characterized in that: The false alarm predictor design and control output form described in "Step 4" are as follows: The output feature vector from the visual language fusion module obtained in step 2 is globally pooled. Each feature vector represents whether a region contains the described target. Finally, a multi-layer perceptron is used to predict whether the target is contained, i.e., 0 / 1 prediction. If the output is binarized by the threshold and the value is 1, the final output is the segmentation mask output by the output graphic decoder and the feature map output by the target region predictor superimposed on the segmentation mask; If the output is binarized to a value of 0, the final output is a target-free segmentation mask.

6. The remote sensing image segmentation method based on cross-modal vision-text fusion according to claim 1, characterized in that: The image decoder design described in "Step 5" is done as follows: The image decoder obtains multi-scale image features from the image encoder, passes the multi-scale image features through the deformable Transformer, obtains multi-scale features with the same dimension as the input, and fuses the high-dimensional features with the low-dimensional features after upsampling, and finally obtains a segmentation mask with the same size as the original image. This segmentation mask is guided by the output of the visual language fusion module to obtain the final predicted segmentation mask.

7. The remote sensing image segmentation method based on cross-modal vision-text fusion according to claim 1, characterized in that: The design of the non-keyword filter and center point predictor described in "Step 6" is as follows: The non-keyword filter has a simple structure. It mainly uses the natural language corpus to extract non-keywords in sentences that contain no or very little semantic information, and also sends the filtered text to the language encoder to obtain its features. The center point predictor receives the features of the original sentence and the filtered sentence as well as the image features output by the image encoder, and performs mutual attention operations on each of them. Finally, it performs fusion decoding to obtain a Gaussian circle prediction map with the center at the target center point. The supervised center point prediction branch formed by these two consecutive operations allows the model to better understand the keywords in the input sentence and improves the model's ability to identify difficult targets through center point prediction.

Images