A remote sensing image reference segmentation method and system

Abstract

The invention discloses a remote sensing image reference segmentation method and system, comprising: inputting an image and a reference text into a reference detection model of the first stage, a text encoding module mapping the reference text into a text sequence, a semantic part-of-speech feature extraction module performing attention weighting on the text sequence to obtain text features fused with text part-of-speech features, an image encoding module performing preliminary image feature extraction on the image, splicing text features and image features and inputting them into a feature extractor to output a detection frame and output features, inputting the output features into a weighted point regression task head, and predicting output key points; taking an initial image as input, a detection frame and key points as double prompts, and inputting a double prompt segmentation model of the second stage, a second stage image encoder encoding the image into an image embedding, a double prompt encoder encoding the double prompts, and a mask decoder performing feature fusion on the image embedding and the double prompt encoding to obtain a final segmentation result. The invention can perform reference segmentation on remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image representative segmentation, and in particular to a remote sensing image representative segmentation method and system. Background Art

[0002] Remote sensing images are surface images and data obtained from drones, airplanes, satellites or other platforms through remote sensing technology, and usually contain a large amount of ground object information. By processing and interpreting remote sensing images and extracting information related to the surface, it is possible to provide large-scale and efficient data support in tasks such as environmental monitoring, resource management, and urban planning. In the fields of smart cities and urban planning, smart transportation, and autonomous driving, it is often necessary to extract and segment specific targets from remote sensing images based on user prompts. For such human-computer interaction tasks, user instructions are often text-based, and the referential segmentation model that integrates text and image information can segment specific targets based on text expressions, which has broad application prospects in remote sensing image analysis.

[0003] The current representative segmentation models are basically designed for natural images, while there are significant differences between remote sensing images and natural images. Since natural images are usually taken at close range, remote sensing images are taken at high altitude and long distance. This leads to the fact that remote sensing images, compared with natural images, have the characteristics of covering a wide geographical space, large differences in the volume of various targets, a large number of small targets, and a large number of similar targets appearing in a single image. Representative segmentation models for natural images, such as those based on the combination of CNN and transformer, MAttNe, etc., mostly use a one-time fusion and segmentation method of text and image features. The model needs to solve the tasks of text image feature fusion, target positioning, and pixel-level segmentation at one time, which makes the model training more difficult. At the same time, this model uses a unified segmentation method for targets of different scales, ignoring the problem that small targets contain less image information than large targets and are more interfered by background information. It is difficult to obtain high accuracy for the representative segmentation of small targets. In addition, this type of model usually uses a single text encoder to extract text features, without effectively utilizing the language characteristics such as part of speech contained in the text itself. The understanding of target nouns and directional adverbs only relies on the fusion with image features, making it difficult to distinguish similar targets at different locations in remote sensing images. Therefore, a referential segmentation method for remote sensing images is urgently needed to improve the segmentation accuracy of remote sensing images. Summary of the invention

[0004] The present invention provides a remote sensing image reference segmentation method and system to solve the problems existing in the above-mentioned prior art. The technical solution is as follows:

[0005] On the one hand, a remote sensing image reference segmentation method is provided, comprising:

[0006] S1, obtain the image and reference text to be segmented;

[0007] S2, input the image and the reference text into the first stage of the reference detection model that has been trained, the reference detection model includes a text encoding module, a semantic part-of-speech feature extraction module, an image encoding module, a Transformer-based feature extractor and a weighted point regression task head, the text encoding module maps the reference text into a text sequence, the semantic part-of-speech feature extraction module performs attention weighting on the text sequence to obtain text features that integrate text part-of-speech features, the image encoding module performs preliminary image feature extraction on the image, splices the text features and image features and inputs them into the feature extractor, further extracts deep text features and image features, outputs a detection box and output features, inputs the output features into the weighted point regression task head, extracts key point features, and predicts output key points;

[0008] S3, taking the initial image as input, taking the detection box and key points as dual prompts, and inputting the trained second-stage dual prompt segmentation model, wherein the dual prompt segmentation model comprises a second-stage image encoder, a dual prompt encoder and a mask decoder, wherein the second-stage image encoder outputs the image encoding as an image embedding, the dual prompt encoder encodes the dual prompt obtained in the first stage, and the mask decoder performs feature fusion on the image embedding and the dual prompt encoding, and predicts the final segmentation result;

[0009] The text encoding module uses byte pair encoding BPE as a text encoder to map the reference text into a text sequence L;

[0010] The semantic part-of-speech feature extraction module uses the natural language processing toolkit NLTK as a semantic part-of-speech feature extractor to extract part-of-speech features from the text, obtains the part-of-speech description {n., a., …, p.} corresponding to each word, and converts it into a part-of-speech sequence through BPE ={ , ,…, }, Represents the maximum text length, using the extracted part-of-speech sequence Perform weighted cross attention with the text sequence L, and set the query weighting link to make the text features more biased towards nouns and adverbs related to location information, and add a hyperparameter greater than 1 and , a learnable query weight for part-of-speech sequences ={ ,…, }, when the text part of speech at the corresponding position is a noun, the query weight of the corresponding position becomes , when the text part of speech at the corresponding position is an adverb, the query weight of the corresponding position becomes , the remaining position weights become , the final query weight is ={ ,…, }, set learnable key weights for the text sequence L Sum weight , after the following cross-attention operation, the semantic part-of-speech enhanced text features are obtained :

[0011]

[0012]

[0013] in are the dimensions of Q and K;

[0014] The image encoding module uses a learnable residual network as the first-stage image encoder to divide the image into n patches and convert each patch into an image feature with a dimension of h;

[0015] The feature extractor is composed of 12 stacked Transformer layers. The detection frame task head and the weighted point regression task head are connected to the output layer of the last Transformer, and the output is a double prompt including a detection frame and key points.

[0016] Optionally, the weighted point regression task head maps the output features to key point prediction values ​​through a linear layer, and generates a size of Gaussian heat map of ,in is the horizontal coordinate of the predicted key point, is the predicted key point ordinate, x and y are the horizontal and vertical coordinates of the point on the Gaussian heat map, h and w are the length and width of the input image, respectively. , use the softmax function to normalize the Gaussian heat map to obtain the weight matrix of each pixel ;

[0017] Set up two spatial matrices X and Y, with the same size as Gaussian thermal Figure 1 To, among which , , , , these two space matrices are The horizontal and vertical coordinate values ​​corresponding to each element in the key point probability distribution are: , find the mathematical expectation of the probability distribution and get the predicted key point coordinates = .

[0018] Optionally, the second stage image encoder uses the ViT model to first change the initial image size to an input form that conforms to the ViT model, and outputs the image embedding ;

[0019] The dual prompt encoder encodes the dual prompt obtained in the first stage, and the process is as follows:

[0020] Since both key points and detection boxes are input into the model in the form of point coordinates, and key points are added as points, the coordinates of the detection box include the upper left corner point of the detection box and the lower right corner point of the detection box, so first directly generate a set of learnable parameters for the coordinate point position , the size is: , where the first dimension represents the point label as the added point, the upper left corner point of the detection box, and the lower right corner point of the detection box;

[0021] Change the detection box to a size of A vector of , where N is the number of detection boxes in each image, is used to treat the detection boxes and key points uniformly, so that all double prompts are converted into point vectors;

[0022] Then, the point position is encoded, the horizontal and vertical coordinates of all point vectors are normalized by adding 0.5, and the normalized point vectors are multiplied by the random Gaussian matrix to obtain the point random Gaussian matrix , the point random Gaussian matrix Multiply each element of Then perform sine function operation and cosine function operation to obtain the point sine matrix , and the point cosine matrix , the point sine matrix With the point cosine matrix Splice and get the point code ;

[0023] Then add the previously generated learnable parameters to the point encoding. , get the double prompt code = + .

[0024] Optionally, the mask decoder embeds the image and double prompt encoding Perform feature fusion and predict the final segmentation result. The process is as follows:

[0025] First, some learnable parameters are still generated, including the size of The mask tensor of and size The iou tensor ,Will , Expanded to the double prompt encoding After the size is consistent, the double prompt encoding Concatenate into a tensor as a double-prompt embedding ;

[0026] The image is embedded It is composed of multiple image features, and generates a full 1 tensor of the same size as the image feature as the image position matrix, which is used to position encode each point of the image feature to obtain the image embedding position code and extend it to embed the image with Consistent size;

[0027] Generate an image embedding with The mask is encoded with the same size and embedded with the image Add together to get the mask image embedding ;

[0028] Embed the mask image , Image Embedding Position Encoding and double prompt embed Input a two-stream Transformer module, which embeds the mask image , Image Embedding Position Encoding and double prompt embed Perform deep feature fusion and output fused mask features And fusion image embedding features ;

[0029] For the fusion mask feature And fusion image embedding features Post-processing, the process is:

[0030] The fused mask feature The first position of the second dimension is taken out as the iou prediction tensor, and the second to fifth positions are taken out as 4 mask prediction tensors, and the fused image is embedded in the feature After two layers of transposed convolution for size transformation, the four mask prediction tensors are respectively input into four independent three-layer multi-layer perceptrons to obtain a mask matching matrix, and the mask matching matrix is ​​embedded in the fusion image feature map. Perform matrix multiplication to obtain the mask candidate matrix, send the iou prediction tensor to the linear layer, and resize it to To match the mask candidate matrix, the iou prediction tensor is used as a weight parameter, a weighted operation is performed on all mask candidate matrices to obtain a mask matrix, and the mask matrix is ​​upsampled to the image size to obtain the final segmentation result.

[0031] Optionally, the two-stream Transformer module is composed of a two-layer two-stream attention mechanism and a cross attention mechanism, and the specific process is as follows:

[0032] 1) In the first layer of the two-stream attention mechanism, the dual prompts are first embedded Perform self-attention operation and normalize to obtain dual prompt features and embed them with the dual prompts Add as query feature ;

[0033] 2) Embed the mask image and image embedding position encoding Addition as a key feature , embed the mask image into As a value feature , cross-attention operation is performed on the above query, key, and value features to obtain the dual prompt embedding Embed the mask image The weighted weight is combined with the query feature Add together to get the initial fused image embedding;

[0034] 3) Use a multi-layer perceptron to transform and normalize the weighted weights to obtain the query features output by the first layer of the two-stream attention mechanism ;

[0035] 4) The query features output by the first layer Embedded with the dual prompt Add as new key feature , the key feature As a new query feature , the query features output by the first layer As a value feature , continue to perform cross-attention operations on the above query, key, and value features and embed them with the mask image After addition, normalization is performed to obtain the key features of the first layer output ;

[0036] 5) The query features output by the first layer of the two-stream attention mechanism and key features As the input of the second layer, the query features Embedded with the dual prompt Add as the initial query feature and the initial key feature, and convert the query feature As the initial value feature, the above initial query, key, and value features are cross-attentioned and combined with the query feature Add the residuals and normalize them to get the new query features of the second layer that fuse the query feature weights output by the first layer. ;

[0037] 6) The key features of the first layer output Mask image embedding as the second layer , replace the mask image embedding of the first layer , repeat steps 2) and 3), and obtain the dual-prompt query feature fused with image embedding through step 3) ;

[0038] 7) Use the dual prompt to query the feature Replace the query features of the first layer output , repeat step 4) to obtain the fused image embedding feature that combines the dual prompt query features ;

[0039] 8) The double prompt query feature and fusion image embedding features Respectively with the double prompt embed and image embedding position encoding Add as fusion query features and fusion key features, and embed the fusion image output by the two-stream attention mechanism into features As the fusion value feature, the cross attention mechanism operation is performed on the above fusion query, key, and value features to obtain a size of Fusion mask features , the fusion mask feature and the fused image embedding feature Together they serve as the output of the two-stream Transformer.

[0040] Optionally, the training process of the reference detection model in the first stage and the dual-prompt segmentation model in the second stage includes:

[0041] Use drones to shoot over multiple locations to obtain multiple images;

[0042] Each image is annotated with a label target box and a mask truth value, and the annotated mask truth values ​​are clustered. Each image obtains multiple cluster center points based on pixel density. , as the true value label of the key point output by the first stage prediction;

[0043] Generate corresponding referential text based on the target characteristics, and divide the obtained image-text pair dataset into training set, validation set and test set;

[0044] The image-text pairs in the training set are used to train the first-stage reference detection model and the second-stage dual-prompt segmentation model, wherein the training of the first-stage reference detection model is:

[0045] In the forward transmission process, the image-text pairs of the training set are loaded, and the output double prompts including the detection box and the key points are obtained according to the data transmission direction in the reference detection model of the first stage. The Smooth L1 function and the mean square error MSE function are used as loss functions to constrain the output, and the Adam optimization is used to update the gradient in the back propagation process;

[0046] The second stage of dual prompt segmentation model training:

[0047] The second-stage model input in the forward transmission process is the image of the training set and the dual prompts obtained in the first stage. During the training of the dual prompt segmentation model, for the second-stage image encoder that is more difficult to train, the ViT model weights are directly loaded and frozen, and only the weights of the dual prompt encoder and mask decoder with fewer parameters are updated. The input image and dual prompt are used to obtain the segmentation results according to the data transmission direction. The DICE function is used as the loss function to constrain the output. Adam optimization is also used to update the gradient of the code verification encoder during the back propagation process.

[0048] In another aspect, a remote sensing image reference segmentation system is provided, the system comprising:

[0049] An acquisition module is used to acquire images and reference texts that need to be segmented;

[0050] A reference detection module, used for inputting the image and the reference text into the first stage of the reference detection model that has been trained, wherein the reference detection model includes a text encoding module, a semantic part-of-speech feature extraction module, an image encoding module, a Transformer-based feature extractor, and a weighted point regression task head, wherein the text encoding module maps the reference text into a text sequence, the semantic part-of-speech feature extraction module performs attention weighting on the text sequence to obtain text features that integrate text part-of-speech features, the image encoding module performs preliminary image feature extraction on the image, concatenates the text features and image features, and inputs them into the feature extractor, further extracts deep text features and image features, outputs a detection box and output features, and inputs the output features into the weighted point regression task head to extract key point features and predict output key points;

[0051] A segmentation module, used to take the initial image as input, the detection box and the key point as dual prompts, and input the trained second-stage dual prompt segmentation model, wherein the dual prompt segmentation model includes a second-stage image encoder, a dual prompt encoder and a mask decoder, wherein the second-stage image encoder outputs the image encoding as an image embedding, the dual prompt encoder encodes the dual prompt obtained in the first stage, and the mask decoder performs feature fusion of the image embedding and the dual prompt encoding to predict the final segmentation result;

[0052] The text encoding module uses byte pair encoding BPE as a text encoder to map the reference text into a text sequence L;

[0053] The semantic part-of-speech feature extraction module uses the natural language processing toolkit NLTK as a semantic part-of-speech feature extractor to extract part-of-speech features from the text, obtains the part-of-speech description {n., a., …, p.} corresponding to each word, and converts it into a part-of-speech sequence through BPE ={ , ,…, }, Represents the maximum text length, using the extracted part-of-speech sequence Perform weighted cross attention with the text sequence L, and set the query weighting link to make the text features more biased towards nouns and adverbs related to location information, and add a hyperparameter greater than 1 and , a learnable query weight for part-of-speech sequences ={ ,…, }, when the text part of speech at the corresponding position is a noun, the query weight of the corresponding position becomes , when the text part of speech at the corresponding position is an adverb, the query weight of the corresponding position becomes , the remaining position weights become , the final query weight is ={ ,…, }, set learnable key weights for the text sequence L Sum weight , after the following cross-attention operation, the semantic part-of-speech enhanced text features are obtained :

[0054]

[0055]

[0056] in are the dimensions of Q and K;

[0057] The image encoding module uses a learnable residual network as the first-stage image encoder to divide the image into n patches and convert each patch into an image feature with a dimension of h;

[0058] The feature extractor is composed of 12 stacked Transformer layers. The detection frame task head and the weighted point regression task head are connected to the output layer of the last Transformer, and the output is a double prompt including a detection frame and key points.

[0059] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above remote sensing image reference segmentation method.

[0060] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above remote sensing image reference segmentation method.

[0061] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0062] 1. By adopting a two-stage algorithm of first detection and then segmentation, small targets in remote sensing images can be segmented more finely, solving the problems of insufficient accuracy of traditional algorithms in processing small targets and difficulty in separating targets from backgrounds. Small targets in complex backgrounds in images can be identified more clearly, significantly improving segmentation accuracy and positioning accuracy.

[0063] 2. Designing a text feature extraction method with deep semantic understanding and language structure modeling can better capture and understand complex semantic information, especially fine-grained language components such as directional adverbs and target nouns, thereby significantly improving the accuracy of text information in target positioning.

[0064] 3. The weighted point regression task head is designed to reduce the model convergence speed of the traditional key point detection task, and the phased training strategy is introduced to avoid the problem of difficult model convergence during two-stage cascade training, thereby enhancing the adaptability and applicability of the model when processing large-scale data.

[0065] 4. Design a dual-cue encoder to perform position encoding on the detection boxes and key points obtained in the first stage. With fewer learnable parameters, the model can understand the position information contained in the dual cues, thereby improving positioning accuracy and effectively saving computing resources.

[0066] 5. Design a mask decoder, make full use of dual-prompt coding to assist in positioning, achieve accurate segmentation within a small range where the target is located, and mine the location information contained in the image features through the image embedding position coding enhancement model. Through the double-stream transformer module based on cross attention, strengthen the guidance of the dual prompts on the segmentation process, and achieve deep fusion of image features and dual-prompt features. Make full use of dual-prompt coding to filter out background information, reduce the difficulty of positioning in the segmentation process, and perform outstandingly in the positioning accuracy of small targets. At the same time, through the post-processing module, the output candidate matrix is ​​weighted using the iou prediction tensor. Compared with directly outputting the segmentation results, it can integrate more global feature information, further optimize the segmentation accuracy, and enhance the adaptability to small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 is a flow chart of a remote sensing image reference segmentation method provided by an embodiment of the present invention;

[0069] Figure 2 It is an overall block diagram of a remote sensing image reference segmentation method provided by an embodiment of the present invention;

[0070] Figure 3 It is a structural block diagram of a semantic part-of-speech feature extraction module provided by an embodiment of the present invention;

[0071] Figure 4 It is a structural block diagram of a sequence-to-sequence reference detection model in the first stage provided by an embodiment of the present invention;

[0072] Figure 5 It is a structural block diagram of a weighted point regression task head provided by an embodiment of the present invention;

[0073] Figure 6 It is a structural block diagram of a dual prompt encoder of a dual prompt segmentation model in the second stage provided by an embodiment of the present invention;

[0074] Figure 7 It is a structural block diagram of a mask decoder of a dual-prompt segmentation model in the second stage provided by an embodiment of the present invention;

[0075] Figure 8 is a structural block diagram of a dual-stream Transformer module provided in an embodiment of the present invention;

[0076] Fig. 9It is a block diagram of a remote sensing image reference segmentation system provided by an embodiment of the present invention;

[0077] Fig.10 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0078] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0079] The embodiment of the present invention provides a remote sensing image reference segmentation method, which can be implemented by an electronic device, and the electronic device can be a terminal or a server. Figure 1 A flow chart of a remote sensing image referent segmentation method is shown, and the processing flow of the method may include the following steps:

[0080] S1, obtain the image and reference text to be segmented;

[0081] S2, inputting the image and the reference text into the first stage reference detection model trained, such as Figure 2 As shown, the reference detection model includes a text encoding module, a semantic part-of-speech feature extraction module, an image encoding module, a Transformer-based feature extractor and a weighted point regression task head. The text encoding module maps the reference text into a text sequence. The semantic part-of-speech feature extraction module performs attention weighting on the text sequence to obtain text features that integrate text part-of-speech features. The image encoding module performs preliminary image feature extraction on the image, splices the text features and image features and inputs them into the feature extractor, further extracts deep text features and image features, outputs a detection box and output features, and inputs the output features into the weighted point regression task head to extract key point features and predict output key points.

[0082] The text encoding module uses Byte Pair Encode (BPE) as a text encoder to map the reference text into a text sequence L;

[0083] The semantic part-of-speech feature extraction module, such as Figure 3 As shown in the figure, the Natural Language Processing Toolkit (NLTK) is used as a semantic part-of-speech feature extractor to extract part-of-speech features from the text, and the part-of-speech description {n., a., ..., p.} corresponding to each word is obtained, such as plane-n., left-a., where n. represents a noun, a. represents an adjective and adverb, and p. represents a preposition, and is converted into a part-of-speech sequence through BPE. ={ , ,…, } (Since the part of speech description is a string constant, and the model can only accept a sequence of numbers), Represents the maximum text length, using the extracted part-of-speech sequence Perform weighted cross attention with the text sequence L, and set the query weighting link to make the text features more biased towards nouns and adverbs related to location information, and add a hyperparameter greater than 1 and , a learnable query weight for part-of-speech sequences ={ ,…, }, when the text part of speech at the corresponding position is a noun, the query weight of the corresponding position becomes , when the text part of speech at the corresponding position is an adverb, the query weight of the corresponding position becomes , the remaining position weights become , the final query weight is ={ ,…, }, set learnable key weights for the text sequence L Sum weight , after the following cross-attention operation, the semantic part-of-speech enhanced text features are obtained :

[0084]

[0085]

[0086] in are the dimensions of Q and K;

[0087] The image encoding module uses a learnable residual network as the first-stage image encoder to divide the image into n patches and convert each patch into an image feature with a dimension of h;

[0088] The feature extractor, such as Figure 4 As shown in FIG. 1 , 12 Transformer layers are stacked, and the detection frame task head and the weighted point regression task head are connected to the output layer of the last Transformer, and a double prompt including a detection frame and key points is output.

[0089] Alternatively, if Figure 5 As shown, the weighted point regression task head maps the output features to key point prediction values ​​through a linear layer, and generates a size of Gaussian heat map of ,in is the horizontal coordinate of the predicted key point, is the predicted key point ordinate, x and y are the horizontal and vertical coordinates of the point on the Gaussian heat map, h and w are the length and width of the input image, respectively. , use the softmax function to normalize the Gaussian heat map to obtain the weight matrix of each pixel ;

[0090] Set up two spatial matrices X and Y, with the same size as Gaussian thermal Figure 1 To, among which , , , , these two space matrices are The horizontal and vertical coordinate values ​​corresponding to each element in the key point probability distribution are: , find the mathematical expectation of the probability distribution and get the predicted key point coordinates = .

[0091] S3. Take the initial image as input, take the detection box and key points as dual prompts, and input them into the trained second-stage dual prompt segmentation model. The dual prompt segmentation model includes a second-stage image encoder, a dual prompt encoder and a mask decoder. The second-stage image encoder outputs the image encoding as an image embedding. The dual prompt encoder encodes the dual prompts obtained in the first stage. The mask decoder performs feature fusion of the image embedding and the dual prompt encoding to predict the final segmentation result.

[0092] Optionally, the second stage image encoder uses the ViT model to first change the initial image size to an input form that conforms to the ViT model, and outputs the image embedding ;

[0093] like Figure 6 As shown, the dual prompt encoder encodes the dual prompt obtained in the first stage, and the process is:

[0094] Since both key points and detection boxes are input into the model in the form of point coordinates, and key points are added as points, the coordinates of the detection box include the upper left corner point of the detection box and the lower right corner point of the detection box, so first directly generate a set of learnable parameters for the coordinate point position , the size is: , where the first dimension represents the point label as the added point, the upper left corner point of the detection box, and the lower right corner point of the detection box;

[0095] Change the detection box to a size of A vector of , where N is the number of detection boxes in each image, is used to treat the detection boxes and key points uniformly, so that all double prompts are converted into point vectors;

[0096] Then, the point position is encoded, the horizontal and vertical coordinates of all point vectors are normalized by adding 0.5, and the normalized point vectors are multiplied by the random Gaussian matrix to obtain the point random Gaussian matrix , the point random Gaussian matrix Multiply each element of Then perform sine function operation and cosine function operation to obtain the point sine matrix , and the point cosine matrix , the point sine matrix With the point cosine matrix Splice and get the point code ;

[0097] Then add the previously generated learnable parameters to the point encoding. , get the double prompt code = + .

[0098] Alternatively, if Figure 7 As shown, the mask decoder embeds the image and double prompt encoding Perform feature fusion and predict the final segmentation result. The process is as follows:

[0099] First, some learnable parameters are still generated, including the size of The mask tensor of and size The iou tensor ,Will , Expanded to the double prompt encoding After the size is consistent, the double prompt encoding Concatenate into a tensor as a double-prompt embedding ;

[0100] The image is embedded It is composed of multiple image features (such as The image embedding consists of 256 ), generates a full-1 tensor of the same size as the image feature as the image position matrix, which is used to position encode each point of the image feature to obtain the image embedding position code and extend it to embed the image with Consistent size;

[0101] Generate an image embedding with The mask is encoded with the same size and embedded with the image Add together to get the mask image embedding ;

[0102] Embed the mask image , Image Embedding Position Encoding and double prompt embed Input a two-stream Transformer module, which embeds the mask image , Image Embedding Position Encoding and double prompt embed Perform deep feature fusion and output fused mask features And fusion image embedding features ;

[0103] For the fusion mask feature And fusion image embedding features Post-processing, the process is:

[0104] The fused mask feature The first position of the second dimension is taken out as the iou prediction tensor, and the second to fifth positions are taken out as 4 mask prediction tensors, and the fused image is embedded in the feature After two layers of transposed convolution for size transformation, the four mask prediction tensors are respectively input into four independent three-layer multi-layer perceptrons to obtain a mask matching matrix, and the mask matching matrix is ​​embedded in the fusion image feature map. Perform matrix multiplication to obtain the mask candidate matrix, send the iou prediction tensor to the linear layer, and resize it to To match the mask candidate matrix, the iou prediction tensor is used as a weight parameter, a weighted operation is performed on all mask candidate matrices to obtain a mask matrix, and the mask matrix is ​​upsampled to the image size to obtain the final segmentation result.

[0105] Alternatively, if Figure 8 As shown in the figure, the two-stream Transformer module consists of a two-layer two-stream attention mechanism and a cross attention mechanism. The specific process is as follows:

[0106] 1) In the first layer of the two-stream attention mechanism, the dual prompts are first embedded Perform self-attention operation and normalize to obtain dual prompt features and embed them with the dual prompts Add as query feature ;

[0107] 2) Embed the mask image and image embedding position encoding Addition as a key feature , embed the mask image into As a value feature , cross-attention operation is performed on the above query, key, and value features to obtain the dual prompt embedding Embed the mask image The weighted weight is combined with the query feature Add together to get the initial fused image embedding;

[0108] 3) Use a multi-layer perceptron to transform and normalize the weighted weights to obtain the query features output by the first layer of the two-stream attention mechanism ;

[0109] 4) The query features output by the first layer Embedded with the dual prompt Add as new key feature , the key feature As a new query feature , the query features output by the first layer As a value feature , continue to perform cross-attention operations on the above query, key, and value features and embed them with the mask image After addition, normalization is performed to obtain the key features of the first layer output ;

[0110] 5) The query features output by the first layer of the two-stream attention mechanism and key features As the input of the second layer, the query features Embedded with the dual prompt Add as the initial query feature and the initial key feature, and convert the query feature As the initial value feature, the above initial query, key, and value features are cross-attentioned and combined with the query feature Add the residuals and normalize them to get the new query features of the second layer that fuse the query feature weights output by the first layer. ;

[0111] 6) The key features of the first layer output Mask image embedding as the second layer , replace the mask image embedding of the first layer , repeat steps 2) and 3), and obtain the dual-prompt query feature fused with image embedding through step 3) ;

[0112] 7) Use the dual prompt to query the feature Replace the query features of the first layer output , repeat step 4) to obtain the fused image embedding feature that combines the dual prompt query features ;

[0113] 8) The double prompt query feature and fusion image embedding features Respectively with the double prompt embed and image embedding position encoding Add as fusion query features and fusion key features, and embed the fusion image output by the two-stream attention mechanism into features As the fusion value feature, the cross attention mechanism operation is performed on the above fusion query, key, and value features to obtain a size of Fusion mask features , the fusion mask feature and the fused image embedding feature Together they serve as the output of the two-stream Transformer.

[0114] Optionally, the training process of the reference detection model in the first stage and the dual-prompt segmentation model in the second stage includes:

[0115] Use a drone to shoot over multiple places to obtain multiple images (the images of the remote sensing data set segmented in the embodiment of the present invention can be shot over multiple places using a drone, and the image categories include common categories such as agriculture, airplanes, baseball fields, beaches, buildings, etc.);

[0116] Label each image with a target box and a true mask value (using Any Label Image software), and cluster the labeled true mask values ​​(using the DBSCAN algorithm in this embodiment of the invention). Each image obtains multiple cluster center points based on pixel density. , as the true value label of the key point output by the first stage prediction;

[0117] Generate corresponding referential text based on target characteristics (the embodiment of the present invention uses ChatGPT4.0), and divide the obtained image-text pair dataset into a training set, a validation set, and a test set;

[0118] The image-text pairs in the training set are used to train the first-stage reference detection model and the second-stage dual-prompt segmentation model, wherein the training of the first-stage reference detection model is:

[0119] In the forward transmission process, the image-text pairs of the training set are loaded, and the output double prompts including the detection box and the key points are obtained according to the data transmission direction in the reference detection model of the first stage. The Smooth L1 function and the mean square error MSE function are used as loss functions to constrain the output, and the Adam optimization is used to update the gradient in the back propagation process;

[0120] The second stage of dual prompt segmentation model training:

[0121] The second-stage model input in the forward transmission process is the image of the training set and the dual prompts obtained in the first stage. During the training of the dual prompt segmentation model, for the second-stage image encoder that is more difficult to train, the ViT model weights are directly loaded and frozen, and only the weights of the dual prompt encoder and mask decoder with fewer parameters are updated. The input image and dual prompt are used to obtain the segmentation results according to the data transmission direction. The DICE function is used as the loss function to constrain the output. Adam optimization is also used to update the gradient of the code verification encoder during the back propagation process.

[0122] like Fig. 9 As shown, an embodiment of the present invention further provides a remote sensing image reference segmentation system, the system comprising:

[0123] An acquisition module 910 is used to acquire an image and a reference text to be segmented;

[0124] The reference detection module 920 is used to input the image and the reference text into the first stage of the reference detection model that has been trained. The reference detection model includes a text encoding module, a semantic part-of-speech feature extraction module, an image encoding module, a Transformer-based feature extractor and a weighted point regression task head. The text encoding module maps the reference text into a text sequence. The semantic part-of-speech feature extraction module performs attention weighting on the text sequence to obtain text features that integrate text part-of-speech features. The image encoding module performs preliminary image feature extraction on the image, splices the text features and image features, and inputs them into the feature extractor to further extract deep text features and image features, output detection boxes and output features, and input the output features into the weighted point regression task head to extract key point features and predict output key points.

[0125] The segmentation module 930 is used to take the initial image as input, the detection box and the key point as dual prompts, and input the trained second-stage dual prompt segmentation model, wherein the dual prompt segmentation model includes a second-stage image encoder, a dual prompt encoder and a mask decoder, wherein the second-stage image encoder outputs the image encoding as an image embedding, the dual prompt encoder encodes the dual prompt obtained in the first stage, and the mask decoder performs feature fusion of the image embedding and the dual prompt encoding to predict the final segmentation result;

[0126] The text encoding module uses byte pair encoding BPE as a text encoder to map the reference text into a text sequence L;

[0127] The semantic part-of-speech feature extraction module uses the natural language processing toolkit NLTK as a semantic part-of-speech feature extractor to extract part-of-speech features from the text, obtains the part-of-speech description {n., a., …, p.} corresponding to each word, and converts it into a part-of-speech sequence through BPE ={ , ,…, }, Represents the maximum text length, using the extracted part-of-speech sequence Perform weighted cross attention with the text sequence L, and set the query weighting link to make the text features more biased towards nouns and adverbs related to location information, and add a hyperparameter greater than 1 and , a learnable query weight for part-of-speech sequences ={ ,…, }, when the text part of speech at the corresponding position is a noun, the query weight of the corresponding position becomes , when the text part of speech at the corresponding position is an adverb, the query weight of the corresponding position becomes , the remaining position weights become , the final query weight is ={ ,…, }, set learnable key weights for the text sequence L Sum weight , after the following cross-attention operation, the semantic part-of-speech enhanced text features are obtained :

[0128]

[0129]

[0130] in are the dimensions of Q and K;

[0131] The image encoding module uses a learnable residual network as the first-stage image encoder to divide the image into n patches and convert each patch into an image feature with a dimension of h;

[0132] The feature extractor is composed of 12 stacked Transformer layers. The detection frame task head and the weighted point regression task head are connected to the output layer of the last Transformer, and the output is a double prompt including a detection frame and key points.

[0133] The functional structure of a remote sensing image reference segmentation system provided by an embodiment of the present invention corresponds to that of a remote sensing image reference segmentation method provided by an embodiment of the present invention, and will not be described in detail here.

[0134] Fig.10 It is a structural diagram of an electronic device 1000 provided in an embodiment of the present invention. The electronic device 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1001 and one or more memories 1002, wherein the memory 1002 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1001 to implement the steps of the above-mentioned remote sensing image reference segmentation method.

[0135] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the remote sensing image reference segmentation method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0136] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image reference segmentation method, characterized in that: The method comprises: S1, obtain the image and reference text to be segmented; S2, input the image and the reference text into the first stage of the reference detection model that has been trained, the reference detection model includes a text encoding module, a semantic part-of-speech feature extraction module, an image encoding module, a Transformer-based feature extractor and a weighted point regression task head, the text encoding module maps the reference text into a text sequence, the semantic part-of-speech feature extraction module performs attention weighting on the text sequence to obtain text features that integrate text part-of-speech features, the image encoding module performs preliminary image feature extraction on the image, splices the text features and image features and inputs them into the feature extractor, further extracts deep text features and image features, outputs a detection box and output features, inputs the output features into the weighted point regression task head, extracts key point features, and predicts output key points; S3, taking the initial image as input, taking the detection box and key points as dual prompts, and inputting the trained second-stage dual prompt segmentation model, wherein the dual prompt segmentation model comprises a second-stage image encoder, a dual prompt encoder and a mask decoder, wherein the second-stage image encoder outputs the image encoding as an image embedding, the dual prompt encoder encodes the dual prompt obtained in the first stage, and the mask decoder performs feature fusion on the image embedding and the dual prompt encoding, and predicts the final segmentation result; The text encoding module uses byte pair encoding BPE as a text encoder to map the reference text into a text sequence L; The semantic part-of-speech feature extraction module uses the natural language processing toolkit NLTK as a semantic part-of-speech feature extractor to extract part-of-speech features from the text, obtains the part-of-speech description {n., a., …, p.} corresponding to each word, and converts it into a part-of-speech sequence through BPE ={ , ,…, }, Represents the maximum text length, using the extracted part-of-speech sequence Perform weighted cross attention with the text sequence L, and set the query weighting link to make the text features more biased towards nouns and adverbs related to location information, and add a hyperparameter greater than 1 and , a learnable query weight for part-of-speech sequences ={ ,…, }, when the text part of speech at the corresponding position is a noun, the query weight of the corresponding position becomes , when the text part of speech at the corresponding position is an adverb, the query weight of the corresponding position becomes , the remaining position weights become , the final query weight is ={ ,…, }, set learnable key weights for the text sequence L Sum weight , after the following cross-attention operation, the semantic part-of-speech enhanced text features are obtained : ； ； in are the dimensions of Q and K; The image encoding module uses a learnable residual network as the first-stage image encoder to divide the image into n patches and convert each patch into an image feature with a dimension of h; The feature extractor is composed of 12 stacked Transformer layers. The detection frame task head and the weighted point regression task head are connected to the output layer of the last Transformer, and the output is a double prompt including a detection frame and key points.

2. The method according to claim 1, characterized in that The weighted point regression task head maps the output features to key point prediction values ​​through a linear layer, and generates a size of Gaussian heat map of ,in is the horizontal coordinate of the predicted key point, The predicted key point ordinate, x and y are the horizontal and vertical coordinates of the point on the Gaussian heat map, h and w are the length and width of the input image, respectively. , use the softmax function to normalize the Gaussian heat map to obtain the weight matrix of each pixel ; Set up two spatial matrices X and Y, with the same size as the Gaussian heat map, where , , , , these two space matrices are The horizontal and vertical coordinate values ​​corresponding to each element in the key point probability distribution are: , find the mathematical expectation of the probability distribution and get the predicted key point coordinates = .

3. The method according to claim 1, characterized in that The second stage image encoder uses the ViT model to first change the initial image size to an input form that conforms to the ViT model, and outputs an image embedding ; The dual prompt encoder encodes the dual prompt obtained in the first stage, and the process is as follows: Since both key points and detection boxes are input into the model in the form of point coordinates, and key points are added as points, the coordinates of the detection box include the upper left corner point of the detection box and the lower right corner point of the detection box, so first directly generate a set of learnable parameters for the coordinate point position , the size is: , where the first dimension represents the point label as the added point, the upper left corner point of the detection box, and the lower right corner point of the detection box; Change the detection box to a size of A vector of is used to treat the detection boxes and key points uniformly, where N is the number of detection boxes in each image, so that all double prompts are converted into point vectors; Then, the point position is encoded, the horizontal and vertical coordinates of all point vectors are normalized by adding 0.5, and the normalized point vectors are multiplied by the random Gaussian matrix to obtain the point random Gaussian matrix , the point random Gaussian matrix Multiply each element of Then perform sine function operation and cosine function operation to obtain the point sine matrix , and the point cosine matrix , the point sine matrix With the point cosine matrix Splice and get the point code ; Then add the previously generated learnable parameters to the point encoding. , get the double prompt code = + .

4. The method according to claim 3, characterized in that: The mask decoder embeds the image and double prompt encoding Perform feature fusion and predict the final segmentation result. The process is as follows: First, some learnable parameters are still generated, including the size of The mask tensor of and size The iou tensor ,Will , Expanded to the double prompt encoding After the size is consistent, the double prompt encoding Concatenate into a tensor as a double-prompt embedding ; The image is embedded It is composed of multiple image features, and generates a full 1 tensor of the same size as the image feature as the image position matrix, which is used to position encode each point of the image feature to obtain the image embedding position code and extend it to embed the image with Consistent size; Generate an image embedding with Encode the mask of the same size and embed it with the image Add together to get the mask image embedding ; Embed the mask image , Image Embedding Position Encoding and double prompt embed Input a two-stream Transformer module, which embeds the mask image , Image Embedding Position Encoding and double prompt embed Perform deep feature fusion and output fused mask features And fusion image embedding features ; For the fusion mask feature And fusion image embedding features Post-processing, the process is: The fused mask feature The first position of the second dimension is taken out as the iou prediction tensor, and the second to fifth positions are taken out as 4 mask prediction tensors, and the fused image is embedded in the feature After two layers of transposed convolution for size transformation, the four mask prediction tensors are respectively input into four independent three-layer multi-layer perceptrons to obtain a mask matching matrix, and the mask matching matrix is ​​embedded in the fusion image feature map. Perform matrix multiplication to obtain the mask candidate matrix, send the iou prediction tensor to the linear layer, and resize it to To match the mask candidate matrix, the iou prediction tensor is used as a weight parameter, a weighted operation is performed on all mask candidate matrices to obtain a mask matrix, and the mask matrix is ​​upsampled to the image size to obtain the final segmentation result.

5. The method according to claim 4, characterized in that The two-stream Transformer module consists of a two-layer two-stream attention mechanism and a cross attention mechanism. The specific process is as follows: 1) In the first layer of the two-stream attention mechanism, the dual prompts are first embedded Perform self-attention operation and normalize to obtain dual prompt features and embed them with the dual prompts Add as query feature ; 2) Embed the mask image and image embedding position encoding Addition as a key feature , embed the mask image into As a value feature , cross-attention operation is performed on the above query, key, and value features to obtain the dual prompt embedding Embed the mask image The weighted weight is combined with the query feature Add together to get the initial fused image embedding; 3) Use a multi-layer perceptron to transform and normalize the weighted weights to obtain the query features output by the first layer of the two-stream attention mechanism ; 4) The query features output by the first layer Embedded with the dual prompt Add as new key feature , the key feature As a new query feature , the query features output by the first layer As a value feature , continue to perform cross-attention operations on the above query, key, and value features and embed them with the mask image After addition, normalization is performed to obtain the key features of the first layer output ; 5) The query features output by the first layer of the two-stream attention mechanism and key features As the input of the second layer, the query features Embedded with the dual prompt Add as the initial query feature and the initial key feature, and convert the query feature As the initial value feature, the above initial query, key, and value features are cross-attentioned and combined with the query feature Add the residuals and normalize them to get the new query features of the second layer that fuse the query feature weights output by the first layer. ; 6) The key features of the first layer output Mask image embedding as the second layer , replace the mask image embedding of the first layer , repeat steps 2) and 3), and obtain the dual-prompt query feature fused with image embedding through step 3) ; 7) Use the dual prompt query feature Replace the query features of the first layer output , repeat step 4) to obtain the fused image embedding feature that combines the dual prompt query features ; 8) The double prompt query feature and fusion image embedding features Respectively with the double prompt embed and image embedding position encoding Add as fusion query features and fusion key features, and embed the fusion image output by the two-stream attention mechanism into features As the fusion value feature, the cross attention mechanism operation is performed on the above fusion query, key, and value features to obtain a size of Fusion mask features , the fusion mask feature and the fused image embedding feature Together they serve as the output of the two-stream Transformer.

6. The method according to claim 1, characterized in that The training process of the first-stage reference detection model and the second-stage dual-cue segmentation model includes: Use drones to shoot over multiple locations to obtain multiple images; Each image is annotated with a label target box and a mask truth value, and the annotated mask truth values ​​are clustered. Each image obtains multiple cluster center points based on pixel density. , as the true value label of the key point output by the first stage prediction; Generate corresponding referential text based on the target characteristics, and divide the obtained image-text pair dataset into training set, validation set and test set; The image-text pairs in the training set are used to train the first-stage reference detection model and the second-stage dual-prompt segmentation model, wherein the training of the first-stage reference detection model is: In the forward transmission process, the image-text pairs of the training set are loaded, and the output double prompts including the detection box and the key points are obtained according to the data transmission direction in the reference detection model of the first stage. The Smooth L1 function and the mean square error MSE function are used as loss functions to constrain the output, and the Adam optimization is used to update the gradient in the back propagation process; The second stage of dual prompt segmentation model training: The second-stage model input in the forward transmission process is the image of the training set and the dual prompts obtained in the first stage. During the training of the dual prompt segmentation model, for the second-stage image encoder that is more difficult to train, the ViT model weights are directly loaded and frozen, and only the weights of the dual prompt encoder and mask decoder with fewer parameters are updated. The input image and dual prompt are used to obtain the segmentation results according to the data transmission direction. The DICE function is used as the loss function to constrain the output. Adam optimization is also used to update the gradient of the code verification encoder during the back propagation process.

7. A remote sensing image reference segmentation system, characterized in that: The system comprises: An acquisition module is used to acquire images and reference texts that need to be segmented; A reference detection module, used for inputting the image and the reference text into the first stage of the reference detection model that has been trained, wherein the reference detection model includes a text encoding module, a semantic part-of-speech feature extraction module, an image encoding module, a Transformer-based feature extractor, and a weighted point regression task head, wherein the text encoding module maps the reference text into a text sequence, the semantic part-of-speech feature extraction module performs attention weighting on the text sequence to obtain text features that integrate text part-of-speech features, the image encoding module performs preliminary image feature extraction on the image, concatenates the text features and image features, and inputs them into the feature extractor, further extracts deep text features and image features, outputs a detection box and output features, and inputs the output features into the weighted point regression task head to extract key point features and predict output key points; A segmentation module, used to take the initial image as input, the detection box and the key point as dual prompts, and input the trained second-stage dual prompt segmentation model, wherein the dual prompt segmentation model includes a second-stage image encoder, a dual prompt encoder and a mask decoder, wherein the second-stage image encoder outputs the image encoding as an image embedding, the dual prompt encoder encodes the dual prompt obtained in the first stage, and the mask decoder performs feature fusion of the image embedding and the dual prompt encoding to predict the final segmentation result; The text encoding module uses byte pair encoding BPE as a text encoder to map the reference text into a text sequence L; The semantic part-of-speech feature extraction module uses the natural language processing toolkit NLTK as a semantic part-of-speech feature extractor to extract part-of-speech features from the text, obtains the part-of-speech description {n., a., …, p.} corresponding to each word, and converts it into a part-of-speech sequence through BPE ={ , ,…, }, Represents the maximum text length, using the extracted part-of-speech sequence Perform weighted cross attention with the text sequence L, and set the query weighting link to make the text features more biased towards nouns and adverbs related to location information, and add a hyperparameter greater than 1 and , a learnable query weight for part-of-speech sequences ={ ,…, }, when the text part of speech at the corresponding position is a noun, the query weight of the corresponding position becomes , when the text part of speech at the corresponding position is an adverb, the query weight of the corresponding position becomes , the remaining position weights become , the final query weight is ={ ,…, }, set learnable key weights for the text sequence L Sum weight , after the following cross-attention operation, the semantic part-of-speech enhanced text features are obtained : ; ; in are the dimensions of Q and K; The image encoding module uses a learnable residual network as the first-stage image encoder to divide the image into n patches and convert each patch into an image feature with a dimension of h; The feature extractor is composed of 12 stacked Transformer layers. The detection frame task head and the weighted point regression task head are connected to the output layer of the last Transformer, and the output is a double prompt including a detection frame and key points.

8. An electronic device, comprising a processor and a memory, wherein at least one instruction is stored in the memory, wherein: The at least one instruction is loaded and executed by the processor to implement the remote sensing image reference segmentation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, wherein at least one instruction is stored in the storage medium, characterized in that: The at least one instruction is loaded and executed by the processor to implement the remote sensing image reference segmentation method as described in any one of claims 1-6.

Images