A cross-modal remote sensing image target segmentation method and device based on open domain

Abstract

The present invention provides a cross-modal remote sensing image target segmentation method and device based on an open domain, and relates to the technical field of remote sensing image segmentation. The method comprises: performing data processing on a large bird's-eye view remote sensing image to obtain a text-guided remote sensing image data set; obtaining foreground label data and background label data based on object mask annotation; performing image segmentation prediction through an open domain image segmentation model according to the text-guided remote sensing image data set to obtain a classification prediction result; optimizing parameters of the open domain image segmentation model according to the classification loss, image feature reconstruction loss, and text feature reconstruction loss to obtain an optimized open domain image segmentation model; inputting the remote sensing image to be segmented and annotated into the optimized open domain image segmentation model to obtain a remote sensing image segmentation result. The present invention is an open cross-modal remote sensing image target segmentation method with flexible segmentation categories and accurate segmentation targets.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image segmentation, and in particular to an open domain-based cross-modal remote sensing image target segmentation method and device. Background Art

[0002] Remote sensing images are surface image data obtained through remote sensing technology. Remote sensing technology mainly uses sensors carried by satellites, aircraft or other aircraft to collect information about surface objects from a long distance. Benefiting from the super-high altitude overlooking shooting distance, remote sensing image imaging results can cover a large area, and accurate identification of imaging results can be widely used in environmental monitoring, resource management, urban planning, agricultural surveys and other fields.

[0003] Unlike the clear boundaries and significant features of objects in natural images, the features between objects in remote sensing images are very fuzzy, and it is difficult for relevant practitioners to directly identify the target objects in remote sensing images. Even with the help of professional image processing tools, the final recognition results will be different due to the differences in specific image processing methods and the subjective understanding of operators. As a result, the analysis of remote sensing images often has a high professional threshold, which greatly limits the development speed of the above-mentioned fields based on remote sensing image analysis.

[0004] In recent years, data-driven modeling methods have gained attention and development because they do not require artificially specified data mapping. Data-driven image processing methods have also begun to receive widespread attention due to their flexibility and robustness. As a typical data-driven model, semantic segmentation models are widely used in the field of image analysis. The main idea of ​​this type of method is to first establish a large-scale dataset with a one-to-one correspondence between image segmentation masks and labels, and then train the model through the dataset to learn the mapping relationship between masks and labels, so as to achieve pixel-level classification of a given image for subsequent analysis. This type of method has made an important contribution to the development of the field of image analysis.

[0005] However, in the actual application and analysis of remote sensing images, traditional semantic segmentation models cannot effectively identify the boundaries of small and scattered objects that are common in remote sensing target images. Typical methods of this type, such as U-Net, DeepLab, etc., only focus on the accurate recognition of the target foreground. However, in remote sensing images, there are often a large number of small and scattered foreground objects, and the boundary between the background and the foreground, that is, the edge of the foreground, is blurred. In this case, the above methods lack effective constraints and supervision information on the background, resulting in poor edge recognition of foreground targets.

[0006] Due to the lack of information outside the image modality, the recognition results of the single-modality model are limited to the data categories seen during training and are not scalable. Affected by the complexity of the environment and the diversity of objects in remote sensing images, it is often the case that the target object does not belong to the category defined in the training set in actual use. In this case, the above model can only re-label and retrain the data, resulting in insufficient model flexibility and increasing the complexity of model updating and maintenance. The existing remote sensing detection model is limited by the fact that the input is only image data, and the model cannot learn the spatial position relationship between objects. In actual use, the resolution of remote sensing images is extremely large, and there are often a large number of targets in the same category, and users often only focus on a part of them. Therefore, after the existing scheme uniformly recognizes all targets of the same category, the user still needs to manually search for the target of interest, which lacks practicality.

[0007] In the existing technology, there is a lack of an open cross-modal remote sensing image target segmentation method with flexible segmentation categories and accurate segmentation targets. Summary of the invention

[0008] In order to solve the technical problems that the existing technology cannot effectively identify small and scattered targets that often appear in remote sensing images; the existing segmentation method can only judge the inherent category and lacks flexibility; and it is difficult to understand the association between objects in the same category and cannot perform detailed processing on the targets in the category, the embodiment of the present invention provides a cross-modal remote sensing image target segmentation method and device based on an open domain. The technical solution is as follows:

[0009] On the one hand, a cross-modal remote sensing image target segmentation method based on an open domain is provided, the method is implemented by a cross-modal remote sensing image target segmentation device, and the method includes:

[0010] A large bird's-eye view remote sensing image of a city is photographed by a drone; data processing is performed on the large bird's-eye view remote sensing image to obtain a text-guided remote sensing image dataset; the text-guided remote sensing image dataset includes a local image, an object text description, and an object mask corresponding to the object text description;

[0011] Based on the object mask, annotating the text-guided remote sensing image dataset to obtain foreground label data and background label data;

[0012] According to the text-guided remote sensing image dataset, image segmentation prediction is performed through an open domain image segmentation model to obtain classification prediction results, image feature reconstruction loss, and text feature reconstruction loss;

[0013] The method of performing image segmentation prediction based on the text-guided remote sensing image dataset by using an open domain image segmentation model to obtain classification prediction results, image feature reconstruction loss, and text feature reconstruction loss includes:

[0014] According to the partial image, position feature encoding is performed by an image encoder to obtain a second image feature and a third image feature;

[0015] According to the text description of the object, encoding text features by a text encoder to obtain a first text feature and a second text feature;

[0016] Based on the first text feature, according to the second image feature, feature fusion is performed by a text-guided image feature fusion module to obtain a fourth image feature;

[0017] Based on the learnable blank feature, according to the first text feature and the fourth image feature, image-text feature alignment is performed through a modal interaction alignment module to obtain aligned image features and panoramic text image features;

[0018] Based on the learnable blank features, according to the aligned image features and the panoramic text image features, a panoramic image decoder is used to perform segmentation image prediction to obtain a classification prediction result;

[0019] According to the third image feature and the fourth image feature, feature reconstruction is performed by an image reconstructor to obtain a reconstructed image feature; based on a mean square error loss function, a loss function is calculated according to the second image feature and the reconstructed image feature to obtain an image feature reconstruction loss;

[0020] According to the second text feature and the fourth image feature, feature reconstruction is performed by a text reconstructor to obtain a reconstructed text feature; based on a mean square error loss function, a loss function is calculated according to the first text feature and the reconstructed text feature to obtain a text feature reconstruction loss;

[0021] Based on a binary cross entropy loss function, a loss function is calculated according to the foreground label data, the background label data and the classification prediction result to obtain a classification loss;

[0022] Optimizing parameters of an open domain image segmentation model according to the classification loss, the image feature reconstruction loss, and the text feature reconstruction loss to obtain an optimized open domain image segmentation model;

[0023] Acquire a remote sensing image to be segmented and labeled; input the remote sensing image to be segmented and labeled into the optimized open domain image segmentation model to obtain a remote sensing image segmentation result.

[0024] On the other hand, a cross-modal remote sensing image target segmentation device based on an open domain is provided, and the device is applied to a cross-modal remote sensing image target segmentation method based on an open domain, and the device comprises:

[0025] A data set construction module is used to take a bird's-eye view remote sensing image of a city through a drone; perform data processing on the bird's-eye view remote sensing image to obtain a text-guided remote sensing image data set; the text-guided remote sensing image data set includes a local image, an object text description, and an object mask corresponding to the object text description;

[0026] A data set annotation module, used to annotate the text-guided remote sensing image data set based on the object mask to obtain foreground label data and background label data;

[0027] A classification prediction module is used to perform image segmentation prediction based on the text-guided remote sensing image dataset through an open domain image segmentation model to obtain classification prediction results, image feature reconstruction loss, and text feature reconstruction loss;

[0028] Wherein, the classification prediction module is further used for:

[0029] According to the partial image, position feature encoding is performed by an image encoder to obtain a second image feature and a third image feature;

[0030] According to the text description of the object, encoding text features by a text encoder to obtain a first text feature and a second text feature;

[0031] Based on the first text feature, according to the second image feature, feature fusion is performed by a text-guided image feature fusion module to obtain a fourth image feature;

[0032] Based on the learnable blank feature, according to the first text feature and the fourth image feature, image-text feature alignment is performed through a modal interaction alignment module to obtain aligned image features and panoramic text image features;

[0033] Based on the learnable blank features, according to the aligned image features and the panoramic text image features, a panoramic image decoder is used to perform segmentation image prediction to obtain a classification prediction result;

[0034] According to the third image feature and the fourth image feature, feature reconstruction is performed by an image reconstructor to obtain a reconstructed image feature; based on a mean square error loss function, a loss function is calculated according to the second image feature and the reconstructed image feature to obtain an image feature reconstruction loss;

[0035] According to the second text feature and the fourth image feature, feature reconstruction is performed by a text reconstructor to obtain a reconstructed text feature; based on a mean square error loss function, a loss function is calculated according to the first text feature and the reconstructed text feature to obtain a text feature reconstruction loss;

[0036] A loss function calculation module, used to calculate the loss function based on the binary cross entropy loss function according to the foreground label data, the background label data and the classification prediction result to obtain the classification loss;

[0037] A model optimization module, used to optimize the parameters of the open domain image segmentation model according to the classification loss, the image feature reconstruction loss and the text feature reconstruction loss, so as to obtain an optimized open domain image segmentation model;

[0038] The image segmentation module is used to obtain the labeled remote sensing image to be segmented; input the labeled remote sensing image to be segmented into the optimized open domain image segmentation model to obtain the remote sensing image segmentation result.

[0039] On the other hand, a cross-modal remote sensing image target segmentation device is provided, and the cross-modal remote sensing image target segmentation device includes: a processor; a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, any one of the above-mentioned cross-modal remote sensing image target segmentation methods based on open domain is implemented.

[0040] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned open domain-based cross-modal remote sensing image target segmentation methods.

[0041] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0042] The present invention proposes a cross-modal remote sensing image target segmentation method based on an open domain. Through an open domain image segmentation model, dual predictions of foreground and background are made, the model's ability to perceive the boundaries of remote sensing targets is enhanced, and its recognition accuracy of small and scattered targets appearing in remote sensing images is enhanced; by introducing text modal data and utilizing the diversity of text, the model's recognition ability is no longer limited to preset categories, but corresponding recognition and segmentation are performed in remote sensing images according to the categories specified by the text, that is, the model obtains the segmentation ability of the open domain; through a modal interaction alignment module, the model is prompted to learn to understand the open domain categories and relative positional relationships that appear in the natural expression process, and to make corresponding segmentation feedback, thereby improving the practicality of the model. The present invention is an open cross-modal remote sensing image target segmentation method with flexible segmentation categories and accurate segmentation targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 It is a flow chart of a cross-modal remote sensing image target segmentation method based on an open domain provided by an embodiment of the present invention;

[0045] Figure 2 It is a block diagram of a cross-modal remote sensing image target segmentation device based on an open domain provided by an embodiment of the present invention;

[0046] Figure 3 It is a structural schematic diagram of a cross-modal remote sensing image target segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0048] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0049] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0050] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0051] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0052] The embodiment of the present invention provides a cross-modal remote sensing image target segmentation method based on an open domain, which can be implemented by a cross-modal remote sensing image target segmentation device, which can be a terminal or a server. Figure 1 The flowchart of the cross-modal remote sensing image target segmentation method based on open domain is shown. The processing flow of the method may include the following steps:

[0053] S1. Use drones to take bird's-eye view remote sensing images of cities; perform data processing on the bird's-eye view remote sensing images to obtain a text-guided remote sensing image dataset; the text-guided remote sensing image dataset includes local images, object text descriptions, and object masks corresponding to the object text descriptions.

[0054] In a feasible implementation, the present invention collects 16 remote sensing color images of different cities through drone photography, each with a resolution of 5616×3744 pixels. Considering the convenience of model training and annotation, the above image data is cropped with a sequential sliding window at a 0.5 overlap rate according to the preset 1200×1200 pixel unit to obtain a local bird's-eye view with local image information, and the above bird's-eye view image is uniformly downsampled to a resolution of 512×512, obtaining a total of 4300 local images.

[0055] Manually annotate the local image; the annotation is divided into two parts, the first part is the text description of the target object in the image, such as "the car next to the red room"; the second part is the object mask annotation of the corresponding target object, that is, the image mask corresponding to the "car" described in the above text. Annotate all data according to the above method to obtain a text-guided remote sensing image dataset containing 4,300 "text description-target mask-local image" ternary data. Divide the data in the dataset into a training set and a verification set in a ratio of 4:1. After the model is trained on the training set, the model results are verified on the verification set.

[0056] S2. Based on the object mask, the text-guided remote sensing image dataset is annotated to obtain foreground label data and background label data.

[0057] Optionally, based on the object mask, the text-guided remote sensing image dataset is annotated to obtain foreground label data and background label data, including:

[0058] According to the object mask annotation, the foreground is annotated at the position where the pixel value is 1 to obtain the foreground label data;

[0059] Based on the foreground label data, the pixel values ​​of the positions other than the foreground label data in the object mask annotation are inverted to obtain the inverted object mask annotation;

[0060] According to the inverted object mask annotation, the background is annotated at the position where the pixel value is 0 to obtain the background label data.

[0061] In a feasible implementation, for the object mask data, the pixel values ​​in the data are only "0" and "1"; the pixel value "1" indicates that the location is the target object, which is the "foreground label". On the basis of retaining the "foreground label" data, the pixel value is inverted to obtain the label of the corresponding pixel value "0" indicating that the location is the background object as the "background label".

[0062] S3, based on the text-guided remote sensing image dataset, image segmentation prediction is performed through an open domain image segmentation model to obtain classification prediction results, image feature reconstruction loss, and text feature reconstruction loss;

[0063] Among them, according to the text-guided remote sensing image dataset, image segmentation prediction is performed through the open domain image segmentation model to obtain classification prediction results, image feature reconstruction loss and text feature reconstruction loss, including:

[0064] According to the local image, position feature encoding is performed by an image encoder to obtain a second image feature and a third image feature;

[0065] According to the text description of the object, a text feature is encoded by a text encoder to obtain a first text feature and a second text feature;

[0066] Based on the first text feature and according to the second image feature, a text-guided image feature fusion module is used to perform feature fusion to obtain a fourth image feature;

[0067] Based on the learnable blank feature, according to the first text feature and the fourth image feature, the image-text feature alignment is performed through the modal interaction alignment module to obtain the aligned image feature and the panoramic text image feature;

[0068] Based on the learnable blank features, the panoramic image decoder is used to perform segmentation image prediction according to the aligned image features and the panoramic text image features to obtain the classification prediction results;

[0069] According to the third image feature and the fourth image feature, feature reconstruction is performed by an image reconstructor to obtain a reconstructed image feature; based on a mean square error loss function, a loss function is calculated according to the second image feature and the reconstructed image feature to obtain an image feature reconstruction loss;

[0070] According to the second text feature and the fourth image feature, feature reconstruction is performed through a text reconstructor to obtain a reconstructed text feature; based on the mean square error loss function, a loss function is calculated according to the first text feature and the reconstructed text feature to obtain a text feature reconstruction loss.

[0071] In a feasible implementation, the present invention designs a cross-modal open domain image segmentation model that can fuse and interpret text information. After encoding the image and text information, the text information is passed through a designed text-guided image feature fusion module, so that the model understands the text content and effectively transfers the target information indicated in the text to the image features for effective fusion of cross-modal information.

[0072] In this process, the text-guided image feature fusion module is optimized by designing image and text reconstruction tasks. The text information and blank information are spliced ​​as the initial panoramic text information, and the text information and the image features of the fused text information output by the above feature fusion module are sent to the panoramic image decoder for cross-modal deep fusion and decoding. The decoding results include foreground prediction results and background prediction results.

[0073] Through the prediction process of "original text + image features fused with text information" to the foreground prediction mask, and the prediction process of "blank text + image features fused with text information" to the background prediction mask, the supervised training of the above two prediction processes is explicitly shown in the results. The model is trained and optimized from both the foreground and background perspectives. The effectiveness of the trained model is verified on the validation set.

[0074] Optionally, according to the local image, position feature encoding is performed by an image encoder to obtain the second image feature and the third image feature, including:

[0075] According to the local image, the uniform grid method is used to perform cropping to obtain the image block;

[0076] Based on a preset function, position relationship encoding is performed on the image block to obtain position encoding of the image block;

[0077] Based on the linear layer, feature extraction is performed on the image block to obtain the image features of the image block;

[0078] According to the image block position coding, feature embedding is performed on the image block image feature to obtain a first image feature;

[0079] According to the first image feature, a visual feature extractor is used to extract a feature to obtain a second image feature; the visual feature extractor is constructed according to a Swin-Transformer model;

[0080] Based on a preset masking ratio, the second image feature is randomly masked to obtain a third image feature.

[0081] In a feasible implementation, the input of the image encoder is local image data. The local image data size is 512×512 pixels. After input, it is cut into 64 image blocks (patches) of 64×64 size in sequence according to the uniform grid method. According to the position of the image patch in the original whole image, the position relationship is encoded to obtain the position encoding corresponding to each patch. The patch data is feature embedded through the linear layer and the position encoding is superimposed to obtain the first image feature.

[0082] The first image feature is input into a visual feature extractor composed of Swin-Transformer to obtain a second image feature F_v. The output feature is randomly masked according to a set masking ratio k to obtain a masked third image feature F_vm.

[0083] Optionally, according to the text description of the object, encoding text features by a text encoder to obtain a first text feature and a second text feature includes:

[0084] Input the object text description into the Chinese text embedder for text embedding to obtain a text vector;

[0085] The text vector is subjected to feature extraction by a text feature extractor to obtain a first text feature; the text feature extractor is constructed based on the BERT model;

[0086] Based on the NLTK text processing tool, perform part-of-speech recognition on the object text description annotation to obtain the text part-of-speech mask;

[0087] Based on the text part-of-speech mask, the first text feature is masked to obtain the second text feature.

[0088] In a feasible implementation, the text encoder inputs the object text description data. The text description data is embedded (Tokenization) by the Chinese text embedder to obtain a text vector; the text vector is input into the text feature extractor composed of the BERT model to obtain the first text feature F_t.

[0089] The text description data is automatically identified by the Natural Language Toolkit (NLTK) text processing tool, and a text mask is generated according to the position of the noun. Among them, the position with a value of "1" indicates that the text at the corresponding position is a noun. The first text feature F_t is masked according to the text mask. The second text feature F_tm after masking is generated, which does not contain the noun feature but contains the remaining position relationship.

[0090] Optionally, based on the first text feature and according to the second image feature, a text-guided image feature fusion module is used to perform feature fusion to obtain a fourth image feature, including:

[0091] Performing context fusion stacking on the second image features to obtain fused context image features;

[0092] Perform matrix product based on the first text feature and the fused context image feature to obtain an affinity attention map;

[0093] Based on the multi-layer perceptron, matrix weighting is performed according to the first text feature and the affinity attention map to obtain the weighted text feature;

[0094] Based on a preset activation function, weighted fusion is performed according to the weighted text feature and the second image feature to obtain a fourth image feature.

[0095] In a feasible implementation, the input in this step is the second image feature F_v and the first text feature F_t. The image feature F_v performs a context fusion operation to promote the perception and fusion of each element in the feature to its surrounding elements. Here, the data size of F_v is 128×128×120 as an example. For each element, a total of 9 adjacent elements including its own 3×3 element centered on it are taken as a piece of data. A total of 128×128×120 pieces of data are obtained, and each piece of data is represented by 9 elements. Represents the element centered on the corresponding element and its surrounding context. Each piece of data maps 9 elements to 1 new element through a linear layer, representing the element of the fused context information; a fused context image feature is obtained, and its data size is 128×128×120.

[0096] The size of the text feature F_t is 768×20. After feature mapping through a linear layer, F_t is matrix-multiplied with the image features of the above contextual spatial information to obtain an affinity attention map with a size of 128×128×768. The value in the attention map represents the degree of correlation between the visual and text at the corresponding position.

[0097] After linear mapping through the multilayer perceptron (MLP), F_t is matrix-multiplied with the above affinity attention map to obtain a weighted text feature map of size 128×128×120, which represents the sensitivity of the image content to the input text. After the intensity map is activated by the activation function, it is weighted to the original image branch to obtain the fourth image feature F_vt referred to by the text.

[0098] It can effectively extract the visually sensitive information of the text without losing the original image features, and enhance the weight of the part of the feature map that has a strong correlation with the object described by the text, so as to achieve the effect of text-guided image feature fusion.

[0099] Optionally, based on the learnable blank feature, according to the first text feature and the fourth image feature, image-text feature alignment is performed through a modal interaction alignment module to obtain aligned image features and panoramic text image features, including:

[0100] Based on the learnable blank feature, the first text feature is concatenated to obtain the third text feature;

[0101] The third text feature is used as the first query variable; based on the first query variable and the cross attention mechanism, the fourth image feature is re-weighted to obtain a weighted text feature;

[0102] Normalizing the weighted text features to obtain normalized text features;

[0103] Based on the self-attention mechanism, the row modality features are fused according to the normalized text features to obtain the fused text features;

[0104] According to the fused text features, dimension mapping is performed through a feedforward neural network to obtain panoramic text image features;

[0105] The fourth image feature is used as the second query variable; based on the second query variable and the cross attention mechanism, the panoramic text image feature is re-weighted to obtain a weighted image feature;

[0106] Normalizing the weighted image features to obtain normalized image features;

[0107] Based on the self-attention mechanism, the row modality features are fused according to the normalized image features to obtain the fused image features;

[0108] According to the fused image features, dimension mapping is performed through a feedforward neural network to obtain the aligned image features.

[0109] In a feasible implementation, the input in this step is a third text feature F_tb including foreground and blank and a fourth image feature F_vt referred to by the text. The alignment of the text and the visual modality is performed in the modality interaction alignment module, including the alignment between the foreground text feature and the feature representing the foreground part in the image feature and the alignment between the blank text feature and the feature representing the background part in the image feature.

[0110] F_tb is used as the first query variable and sent to the cross-attention layer with F_vt for interaction and reweighting between features. After the result is normalized, the inter-modal features are further interacted and fused through the self-attention layer, and dimension mapping is performed in the feedforward neural network layer to obtain the panoramic text image features representing "target & background and text". Repeat the above process, use F_vt as the second query variable, interact and align with the panoramic text image features representing "target & background and text", and obtain the aligned image features.

[0111] Optionally, based on the learnable blank features, according to the aligned image features and the panoramic text image features, the panoramic image decoder performs segmentation image prediction to obtain a classification prediction result, including:

[0112] Based on the preset learnable blank features, the panoramic text image features are reversely split to obtain the foreground text features and the background text features;

[0113] Based on the multi-layer perceptron, foreground prediction is performed according to the aligned image features and foreground text features to obtain the foreground prediction result;

[0114] Based on the multi-layer perceptron, background prediction is performed according to the aligned image features and background text features to obtain the background prediction result.

[0115] In a feasible implementation, the text features are reversely split according to the previous splicing method to obtain text features representing the target object and the background respectively, and are compressed according to the feature dimensions respectively to obtain one feature representing the foreground and one feature representing the background respectively.

[0116] After the aligned image features are dimensionally adjusted through the MLP layer, matrix multiplication is performed with the two text features to obtain two prediction results with the same size as the original image, representing the prediction of the foreground target and the prediction of the background target, respectively.

[0117] In a feasible implementation, the reconstructor is divided into two parts: an image reconstructor and a text reconstructor. For the image reconstructor, the input is the third image feature F_vm and the fourth image feature F_vt; and the input of the text reconstructor is the second text feature F_tm and the fourth image feature F_vt.

[0118] The reconstruction is completed by guiding the masked image or text features through F_vt, which contains both complete image and text information. F_vm or F_tm and F_vt are sent to the cross-attention layer for shallow feature fusion. The fusion result is normalized and then passed through the self-attention layer for deep feature fusion. After the result is normalized, the final reconstruction result is output through the feedforward neural network, which is recorded as the reconstructed image feature F_vc and the reconstructed text feature F_tc. The loss is calculated between the reconstruction result and the corresponding encoder output feature. The loss function uses the mean squared error (MSE). The loss value is used to measure the similarity between the reconstructed feature and the original feature. The specific loss function formula is as follows (1) and (2):

[0119] (1);

[0120] (2);

[0121] Among them, l2 represents the image feature reconstruction loss, l3 represents the text feature reconstruction loss, and N represents the total number of features.

[0122] Since the classification prediction result is obtained by repeatedly fusing the data of the corresponding modality with the fourth image feature F_vt containing complete information, the optimization process essentially guides F_vt to optimize the extraction of more complete and clear image text features. This alleviates the information loss problem that occurs in the original guided fusion process of modal features, improves the model feature fusion effect, and thus enhances the model capability.

[0123] S4. Based on the binary cross entropy loss function, the loss function is calculated according to the foreground label data, the background label data and the classification prediction results to obtain the classification loss.

[0124] In a feasible implementation, after obtaining the foreground prediction results and the background prediction results, the loss is calculated with the foreground label data and the background label data respectively. The accuracy of the prediction results is measured by the loss value, and the model is optimized. Considering that there are only two types of labels, the binary cross-entropy loss (BCE) is selected as the loss function, and the classification loss function The calculation process is as follows:

[0125] (3);

[0126] (4);

[0127] (5);

[0128] Among them, BCELoss frepresents the loss between the foreground prediction and the foreground label, BCELoss b represents the loss between background prediction and background label; γ1 is the weight of foreground prediction, which is empirically taken as 0.6; γ2 is the weight of background prediction, which is empirically taken as 0.4; P represents the number of pixels in the image; G i Indicates the label of the corresponding pixel, with a value of 0 or 1; Pred i Represents the classification prediction result, which is a probability value between 0 and 1. The smaller the loss value, the closer the prediction result is to the label, and the smaller the parameter amplitude that the model needs to update, and vice versa, the larger the amplitude that needs to be updated.

[0129] By explicitly supervising the foreground and background predictions separately, the model focuses on the prediction effects of both the target foreground and background. For small and scattered targets commonly found in remote sensing images, this design can more effectively learn the difference between foreground and background, capture difficult-to-distinguish edge areas, and thus reduce the interference of background on model learning and improve model recognition accuracy.

[0130] S5. According to the classification loss, the image feature reconstruction loss and the text feature reconstruction loss, the parameters of the open domain image segmentation model are optimized to obtain the optimized open domain image segmentation model.

[0131] In a feasible implementation, the main loss of remote sensing image segmentation is the classification loss between the predicted mask and the real target mask, which is used to directly optimize the model to complete the given target segmentation task. The mask reconstruction loss of the image and text is used to assist in optimizing the feature extraction fusion to further improve the model effect.

[0132] S6. Obtain a remote sensing image to be segmented and labeled; input the remote sensing image to be segmented and labeled into the optimized open domain image segmentation model to obtain a remote sensing image segmentation result.

[0133] In a feasible implementation, the panoramic prediction method and the corresponding panoramic decoder adopted by the present invention significantly enhance the model's perception of edge information that is difficult to capture in remote sensing images, thereby improving the model's recognition accuracy for targets that often appear to be "small and scattered". Text modal information is introduced to break the limitation that the original single image modal model can only classify inherent categories. A text-guided feature fusion module is designed to fuse and extract corresponding information, guiding the model to acquire the ability to understand positional relationships within a class. The model can identify and segment specific targets within a class based on text descriptions. Without changing the original model structure. By designing reconstruction tasks, the model's internal feature extraction capabilities are effectively improved. Thereby improving the model recognition accuracy.

[0134] The present invention proposes a cross-modal remote sensing image target segmentation method based on an open domain. Through an open domain image segmentation model, dual predictions of foreground and background are made, the model's ability to perceive the boundaries of remote sensing targets is enhanced, and its recognition accuracy of small and scattered targets appearing in remote sensing images is enhanced; by introducing text modal data and utilizing the diversity of text, the model's recognition ability is no longer limited to preset categories, but corresponding recognition and segmentation are performed in remote sensing images according to the categories specified by the text, that is, the model obtains the segmentation ability of the open domain; through a modal interaction alignment module, the model is prompted to learn to understand the open domain categories and relative positional relationships that appear in the natural expression process, and to make corresponding segmentation feedback, thereby improving the practicality of the model. The present invention is an open cross-modal remote sensing image target segmentation method with flexible segmentation categories and accurate segmentation targets.

[0135] Figure 2 The present invention is a block diagram of a cross-modal remote sensing image target segmentation device based on an open domain according to an exemplary embodiment. The device is used in a cross-modal remote sensing image target segmentation method based on an open domain. Figure 2 The device includes a data set construction module 210, a data set annotation module 220, a classification prediction module 230, a loss function calculation module 240, a model optimization module 250 and an image segmentation module 260. Among them:

[0136] The data set construction module 210 is used to take a bird's-eye view remote sensing image of a city through a drone; perform data processing on the bird's-eye view remote sensing image to obtain a text-guided remote sensing image data set; the text-guided remote sensing image data set includes a local image, an object text description, and an object mask corresponding to the object text description;

[0137] A data set annotation module 220 is used to annotate the text-guided remote sensing image data set based on the object mask to obtain foreground label data and background label data;

[0138] The classification prediction module 230 is used to perform image segmentation prediction based on the text-guided remote sensing image dataset through an open domain image segmentation model to obtain classification prediction results, image feature reconstruction loss, and text feature reconstruction loss;

[0139] The classification prediction module 230 is further used for:

[0140] According to the local image, position feature encoding is performed by an image encoder to obtain a second image feature and a third image feature;

[0141] According to the text description of the object, a text feature is encoded by a text encoder to obtain a first text feature and a second text feature;

[0142] Based on the first text feature and according to the second image feature, a text-guided image feature fusion module is used to perform feature fusion to obtain a fourth image feature;

[0143] Based on the learnable blank feature, according to the first text feature and the fourth image feature, the image-text feature alignment is performed through the modal interaction alignment module to obtain the aligned image feature and the panoramic text image feature;

[0144] Based on the learnable blank features, the panoramic image decoder is used to perform segmentation image prediction according to the aligned image features and the panoramic text image features to obtain the classification prediction results;

[0145] According to the third image feature and the fourth image feature, feature reconstruction is performed by an image reconstructor to obtain a reconstructed image feature; based on a mean square error loss function, a loss function is calculated according to the second image feature and the reconstructed image feature to obtain an image feature reconstruction loss;

[0146] According to the second text feature and the fourth image feature, feature reconstruction is performed by a text reconstructor to obtain a reconstructed text feature; based on a mean square error loss function, a loss function is calculated according to the first text feature and the reconstructed text feature to obtain a text feature reconstruction loss;

[0147] A loss function calculation module 240 is used to calculate the loss function based on the binary cross entropy loss function according to the foreground label data, the background label data and the classification prediction result to obtain the classification loss;

[0148] A model optimization module 250 is used to optimize the parameters of the open domain image segmentation model according to the classification loss, the image feature reconstruction loss and the text feature reconstruction loss to obtain an optimized open domain image segmentation model;

[0149] The image segmentation module 260 is used to obtain the labeled remote sensing image to be segmented; input the labeled remote sensing image to be segmented into the optimized open domain image segmentation model to obtain the remote sensing image segmentation result.

[0150] Optionally, the data set annotation module 220 is further configured to:

[0151] According to the object mask annotation, the foreground is annotated at the position where the pixel value is 1 to obtain the foreground label data;

[0152] Based on the foreground label data, the pixel values ​​of the positions other than the foreground label data in the object mask annotation are inverted to obtain the inverted object mask annotation;

[0153] According to the inverted object mask annotation, the background is annotated at the position where the pixel value is 0 to obtain the background label data.

[0154] Optionally, the classification prediction module 230 is further used to:

[0155] According to the local image, the uniform grid method is used to perform cropping to obtain the image block;

[0156] Based on a preset function, position relationship encoding is performed on the image block to obtain position encoding of the image block;

[0157] Based on the linear layer, feature extraction is performed on the image block to obtain the image features of the image block;

[0158] According to the image block position coding, feature embedding is performed on the image block image feature to obtain a first image feature;

[0159] According to the first image feature, a visual feature extractor is used to extract a feature to obtain a second image feature; the visual feature extractor is constructed according to a Swin-Transformer model;

[0160] Based on a preset masking ratio, the second image feature is randomly masked to obtain a third image feature.

[0161] Optionally, the classification prediction module 230 is further used to:

[0162] Input the object text description into the Chinese text embedder for text embedding to obtain a text vector;

[0163] The text vector is subjected to feature extraction by a text feature extractor to obtain a first text feature; the text feature extractor is constructed based on the BERT model;

[0164] Based on the NLTK text processing tool, perform part-of-speech recognition on the object text description annotation to obtain the text part-of-speech mask;

[0165] Based on the text part-of-speech mask, the first text feature is masked to obtain the second text feature.

[0166] Optionally, the classification prediction module 230 is further used to:

[0167] Performing context fusion stacking on the second image features to obtain fused context image features;

[0168] Perform matrix product based on the first text feature and the fused context image feature to obtain an affinity attention map;

[0169] Based on the multi-layer perceptron, matrix weighting is performed according to the first text feature and the affinity attention map to obtain the weighted text feature;

[0170] Based on a preset activation function, weighted fusion is performed according to the weighted text feature and the second image feature to obtain a fourth image feature.

[0171] Optionally, the classification prediction module 230 is further used to:

[0172] Based on the learnable blank feature, the first text feature is concatenated to obtain the third text feature;

[0173] The third text feature is used as the first query variable; based on the first query variable and the cross attention mechanism, the fourth image feature is re-weighted to obtain a weighted text feature;

[0174] Normalizing the weighted text features to obtain normalized text features;

[0175] Based on the self-attention mechanism, the row modality features are fused according to the normalized text features to obtain the fused text features;

[0176] According to the fused text features, dimension mapping is performed through a feedforward neural network to obtain panoramic text image features;

[0177] The fourth image feature is used as the second query variable; based on the second query variable and the cross attention mechanism, the panoramic text image feature is re-weighted to obtain a weighted image feature;

[0178] Normalizing the weighted image features to obtain normalized image features;

[0179] Based on the self-attention mechanism, the row modality features are fused according to the normalized image features to obtain the fused image features;

[0180] According to the fused image features, dimension mapping is performed through a feedforward neural network to obtain the aligned image features.

[0181] Optionally, the classification prediction module 230 is further used to:

[0182] Based on the preset learnable blank features, the panoramic text image features are reversely split to obtain the foreground text features and the background text features;

[0183] Based on the multi-layer perceptron, foreground prediction is performed according to the aligned image features and foreground text features to obtain the foreground prediction result;

[0184] Based on the multi-layer perceptron, background prediction is performed according to the aligned image features and background text features to obtain the background prediction result.

[0185] The present invention proposes a cross-modal remote sensing image target segmentation method based on an open domain. Through an open domain image segmentation model, dual predictions of foreground and background are made, the model's ability to perceive the boundaries of remote sensing targets is enhanced, and its recognition accuracy of small and scattered targets appearing in remote sensing images is enhanced; by introducing text modal data and utilizing the diversity of text, the model's recognition ability is no longer limited to preset categories, but corresponding recognition and segmentation are performed in remote sensing images according to the categories specified by the text, that is, the model obtains the segmentation ability of the open domain; through a modal interaction alignment module, the model is prompted to learn to understand the open domain categories and relative positional relationships that appear in the natural expression process, and to make corresponding segmentation feedback, thereby improving the practicality of the model. The present invention is an open cross-modal remote sensing image target segmentation method with flexible segmentation categories and accurate segmentation targets.

[0186] Figure 3 is a schematic diagram of the structure of a cross-modal remote sensing image target segmentation device provided by an embodiment of the present invention, such as Figure 3 As shown, the cross-modal remote sensing image target segmentation device may include the above Figure 2 The cross-modal remote sensing image target segmentation device based on open domain is shown. Optionally, the cross-modal remote sensing image target segmentation device 310 may include a first processor 2001 .

[0187] Optionally, the cross-modal remote sensing image target segmentation device 310 may further include a memory 2002 and a transceiver 2003 .

[0188] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0189] Combine the following Figure 3 The components of the cross-modal remote sensing image target segmentation device 310 are specifically introduced as follows:

[0190] The first processor 2001 is the control center of the cross-modal remote sensing image target segmentation device 310, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0191] Optionally, the first processor 2001 may perform various functions of the cross-modal remote sensing image target segmentation device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0192] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0193] In a specific implementation, as an embodiment, the cross-modal remote sensing image target segmentation device 310 may also include multiple processors, such as Figure 3 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0194] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0195] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0196] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0197] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0198] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and communicate with the first processor 2001 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0199] It should be noted that Figure 3 The structure of the cross-modal remote sensing image target segmentation device 310 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0200] In addition, the technical effects of the cross-modal remote sensing image target segmentation device 310 can refer to the technical effects of the cross-modal remote sensing image target segmentation method based on the open domain described in the above method embodiment, and will not be repeated here.

[0201] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0202] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0203] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0204] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0205] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0206] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0207] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0208] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0209] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0212] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0213] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A cross-modal remote sensing image target segmentation method based on open domain, characterized in that: The method comprises: A large bird's-eye view remote sensing image of a city is photographed by a drone; data processing is performed on the large bird's-eye view remote sensing image to obtain a text-guided remote sensing image dataset; the text-guided remote sensing image dataset includes a local image, an object text description, and an object mask corresponding to the object text description; Based on the object mask, annotating the text-guided remote sensing image dataset to obtain foreground label data and background label data; According to the text-guided remote sensing image dataset, image segmentation prediction is performed through an open domain image segmentation model to obtain classification prediction results, image feature reconstruction loss and text feature reconstruction loss, including: According to the partial image, position feature encoding is performed by an image encoder to obtain a second image feature, and based on a preset masking ratio, the second image feature is randomly masked to obtain a third image feature; According to the text description of the object, a text feature is encoded by a text encoder to obtain a first text feature, and the first text feature is masked based on a text part-of-speech mask to obtain a second text feature; Based on the first text feature, according to the second image feature, feature fusion is performed by a text-guided image feature fusion module to obtain a fourth image feature; Based on the learnable blank feature, according to the first text feature and the fourth image feature, image-text feature alignment is performed through a modal interaction alignment module to obtain aligned image features and panoramic text image features; Based on the learnable blank features, according to the aligned image features and the panoramic text image features, a panoramic image decoder is used to perform segmentation image prediction to obtain a classification prediction result; According to the third image feature and the fourth image feature, feature reconstruction is performed by an image reconstructor to obtain a reconstructed image feature; based on a mean square error loss function, a loss function is calculated according to the second image feature and the reconstructed image feature to obtain an image feature reconstruction loss; According to the second text feature and the fourth image feature, feature reconstruction is performed by a text reconstructor to obtain a reconstructed text feature; based on a mean square error loss function, a loss function is calculated according to the first text feature and the reconstructed text feature to obtain a text feature reconstruction loss; Based on a binary cross entropy loss function, a loss function is calculated according to the foreground label data, the background label data and the classification prediction result to obtain a classification loss; Optimizing parameters of an open domain image segmentation model according to the classification loss, the image feature reconstruction loss, and the text feature reconstruction loss to obtain an optimized open domain image segmentation model; Acquire a remote sensing image to be segmented and labeled; input the remote sensing image to be segmented and labeled into the optimized open domain image segmentation model to obtain a remote sensing image segmentation result.

2. The open domain based cross-modal remote sensing image target segmentation method according to claim 1, characterized in that: The step of labeling the text-guided remote sensing image data set based on the object mask to obtain foreground label data and background label data includes: According to the object mask, a foreground mark is performed on a position where the pixel value is 1 to obtain foreground label data; Based on the foreground label data, invert the pixel values ​​of the positions other than the foreground label data in the object mask annotation to obtain the inverted object mask annotation; According to the inverted object mask annotation, background annotation is performed on positions where the pixel value is 0 to obtain background label data.

3. The open domain based cross-modal remote sensing image target segmentation method according to claim 1, characterized in that: The step of encoding the position feature of the partial image by an image encoder to obtain a second image feature, and randomly masking the second image feature based on a preset masking ratio to obtain a third image feature includes: According to the local image, a uniform grid method is used to perform a cropping process to obtain an image block; Based on a preset function, encoding the position relationship of the image block to obtain an image block position code; Based on the linear layer, feature extraction is performed on the image block to obtain image features of the image block; According to the image block position code, feature embedding is performed on the image feature of the image block to obtain a first image feature; According to the first image feature, feature extraction is performed by a visual feature extractor to obtain a second image feature; the visual feature extractor is constructed based on a Swin-Transformer model.

4. The open domain based cross-modal remote sensing image target segmentation method according to claim 1, characterized in that: The step of encoding text features by a text encoder according to the text description of the object to obtain a first text feature, and masking the first text feature based on a text part-of-speech mask to obtain a second text feature includes: Inputting the object text description into a Chinese text embedder for text embedding to obtain a text vector; Performing feature extraction on the text vector through a text feature extractor to obtain a first text feature; the text feature extractor is constructed according to a BERT model; Based on the NLTK text processing tool, part-of-speech recognition is performed on the object text description annotation to obtain a text part-of-speech mask.

5. The open domain based cross-modal remote sensing image target segmentation method according to claim 1, characterized in that: The step of performing feature fusion based on the first text feature and the second image feature through a text-guided image feature fusion module to obtain a fourth image feature includes: Performing context fusion and stacking on the second image features to obtain fused context image features; Perform matrix product according to the first text feature and the fused context image feature to obtain an affinity attention map; Based on a multi-layer perceptron, matrix weighting is performed according to the first text feature and the affinity attention map to obtain a weighted text feature; Based on a preset activation function, weighted fusion is performed on the weighted text feature and the second image feature to obtain a fourth image feature.

6. The open domain based cross-modal remote sensing image target segmentation method according to claim 1, characterized in that: The method of performing image-text feature alignment based on the learnable blank feature according to the first text feature and the fourth image feature through a modal interaction alignment module to obtain aligned image features and panoramic text image features includes: Based on the learnable blank feature, performing feature concatenation on the first text feature to obtain a third text feature; Using the third text feature as a first query variable; based on the first query variable and a cross-attention mechanism, re-weighting according to the fourth image feature to obtain a weighted text feature; Normalizing the weighted text features to obtain normalized text features; Based on the self-attention mechanism, modal feature fusion is performed according to the normalized text features to obtain fused text features; According to the fused text features, dimension mapping is performed through a feedforward neural network to obtain panoramic text image features; The fourth image feature is used as a second query variable; based on the second query variable and the cross attention mechanism, the panoramic text image feature is re-weighted to obtain a weighted image feature; Normalizing the weighted image features to obtain normalized image features; Based on the self-attention mechanism, modal feature fusion is performed according to the normalized image features to obtain fused image features; According to the fused image features, dimension mapping is performed through a feedforward neural network to obtain aligned image features.

7. The open domain based cross-modal remote sensing image target segmentation method according to claim 1, characterized in that: The method of performing segmentation image prediction based on the learnable blank feature and the aligned image feature and the panoramic text image feature through a panoramic image decoder to obtain a classification prediction result includes: Based on preset learnable blank features, reverse splitting the panoramic text image features to obtain foreground text features and background text features; Based on a multi-layer perceptron, foreground prediction is performed according to the aligned image features and the foreground text features to obtain a foreground prediction result; Based on a multi-layer perceptron, background prediction is performed according to the aligned image features and the background text features to obtain a background prediction result.

8. A cross-modal remote sensing image target segmentation device based on an open domain, wherein the cross-modal remote sensing image target segmentation device based on an open domain is used to implement the cross-modal remote sensing image target segmentation method based on an open domain as claimed in any one of claims 1 to 7, characterized in that: The device comprises: A data set construction module is used to take a bird's-eye view remote sensing image of a city through a drone; perform data processing on the bird's-eye view remote sensing image to obtain a text-guided remote sensing image data set; the text-guided remote sensing image data set includes a local image, an object text description, and an object mask corresponding to the object text description; A data set annotation module, used to annotate the text-guided remote sensing image data set based on the object mask to obtain foreground label data and background label data; A classification prediction module is used to perform image segmentation prediction based on the text-guided remote sensing image dataset through an open domain image segmentation model to obtain classification prediction results, image feature reconstruction loss, and text feature reconstruction loss; Wherein, the classification prediction module is further used for: According to the partial image, position feature encoding is performed by an image encoder to obtain a second image feature and a third image feature; According to the text description of the object, encoding text features by a text encoder to obtain a first text feature and a second text feature; Based on the first text feature, according to the second image feature, feature fusion is performed by a text-guided image feature fusion module to obtain a fourth image feature; Based on the learnable blank feature, according to the first text feature and the fourth image feature, image-text feature alignment is performed through a modal interaction alignment module to obtain aligned image features and panoramic text image features; Based on the learnable blank features, according to the aligned image features and the panoramic text image features, a panoramic image decoder is used to perform segmentation image prediction to obtain a classification prediction result; According to the third image feature and the fourth image feature, feature reconstruction is performed by an image reconstructor to obtain a reconstructed image feature; based on a mean square error loss function, a loss function is calculated according to the second image feature and the reconstructed image feature to obtain an image feature reconstruction loss; According to the second text feature and the fourth image feature, feature reconstruction is performed by a text reconstructor to obtain a reconstructed text feature; based on a mean square error loss function, a loss function is calculated according to the first text feature and the reconstructed text feature to obtain a text feature reconstruction loss; A loss function calculation module, used to calculate the loss function based on the binary cross entropy loss function according to the foreground label data, the background label data and the classification prediction result to obtain the classification loss; A model optimization module, used to optimize the parameters of the open domain image segmentation model according to the classification loss, the image feature reconstruction loss and the text feature reconstruction loss, so as to obtain an optimized open domain image segmentation model; The image segmentation module is used to obtain the labeled remote sensing image to be segmented; input the labeled remote sensing image to be segmented into the optimized open domain image segmentation model to obtain the remote sensing image segmentation result.

9. A cross-modal remote sensing image target segmentation device, characterized in that: The cross-modal remote sensing image target segmentation device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images