Remote sensing image change area detection method and device based on multimodal prompt guidance

Abstract

The present application discloses a remote sensing image change region detection method and device based on multimodal prompt guidance, the method comprising: constructing a multimodal remote sensing image change detection dataset corresponding to the change detection field, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data and semantic category data of the change region; establishing an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture; training the initial remote sensing image change detection network through the multimodal remote sensing image change detection dataset, and determining the remote sensing image change detection network based on the acquired model parameters; inputting the multimodal prompt into the remote sensing image change detection network to obtain the change detection result of the remote sensing image class of interest.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and more specifically, to a remote sensing image change area detection method and device based on multi-modal prompt guidance. Background Art

[0002] With the rapid development of computer vision technology, deep learning has been widely used in various fields of remote sensing image processing, such as classification, denoising, image generation and fusion, target detection, etc. Deep learning has the characteristics of human reasoning and has a natural advantage in feature extraction in change detection, and has become an important means of remote sensing change detection. Remote sensing change detection methods based on deep learning usually extract deep features of remote sensing images and their differences through neural networks, train change detection models under the guidance of learning strategies, and output results according to detection schemes. This type of method has high accuracy and strong robustness, and has developed into the mainstream of current remote sensing image change detection research.

[0003] However, with the rapid advancement of aerospace technology, remote sensing data has become more convenient to obtain, and users' needs for change information have become more diverse, extending from simple judgment of whether there is a change to the identification of multiple types of changes (such as increased buildings, reduced vegetation, and expanded water bodies). These diverse needs are often accompanied by changes in data resolution, color tone, scene, and other factors. The current solution is usually to design a special detection network for specific data types and application scenarios, and collect a large number of training images for annotation and training. This method is time-consuming and labor-intensive, with high annotation costs and long cycles. Therefore, it is of great significance to study a change detection method that is suitable for multiple scenarios, has strong generalization and efficiency, so that it can still maintain high detection accuracy and achieve out-of-the-box capabilities in different scenarios, in order to promote production applications. Summary of the invention

[0004] In response to at least one defect or improvement need in the prior art, the present invention provides a method and device for detecting changed areas in remote sensing images based on multimodal prompt guidance, which combines the stability and generalization of the Vision Transformer visual encoder in the field of computer vision and the large language model in the field of natural language processing in feature extraction and high-level semantic feature processing. Through the detector-guidance remote sensing image change detection framework, the attention of change detection is focused on the region of interest corresponding to the prompt text, thereby achieving stable and universal remote sensing image change detection.

[0005] To achieve the above-mentioned purpose, according to a first aspect of the present invention, a method for detecting changed areas in remote sensing images based on multimodal prompt guidance is provided, the method comprising: constructing a multimodal remote sensing image change detection dataset corresponding to the change detection field, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data and semantic category data of the changed area; establishing an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture; training the initial remote sensing image change detection network through the multimodal remote sensing image change detection dataset, and determining the remote sensing image change detection network based on the acquired model parameters; inputting the multimodal prompt into the remote sensing image change detection network to obtain the change detection result of the remote sensing image class of interest.

[0006] In an exemplary embodiment, the construction of a multimodal remote sensing image change detection dataset corresponding to the change detection field includes: determining the collected change detection dataset including the remote sensing image data, the change label range data and the semantic category data of the change area as the multimodal remote sensing image change detection dataset; and processing the remote sensing image data into a binary image according to the change label format type corresponding to the multimodal remote sensing image change detection dataset.

[0007] In an exemplary embodiment, the establishment of an initial remote sensing image change detection network based on multimodal prompt guidance includes: the initial remote sensing image change detection network includes a feature extraction module, a guide module and a detector module; the feature extraction module is composed of an image feature extraction module and a text feature extraction module, the image feature extraction module in the feature extraction module is used to extract features from the input dual-phase image, and encode them into dual-phase image features, and the text feature extraction module in the feature extraction module is used to extract features from the input semantic guidance, and encode them into guidance features; the dual-phase image features are respectively transmitted to the guide module and the detector module, and the guidance features are transmitted to the guide module.

[0008] In an exemplary embodiment, the dual-phase image features are respectively transmitted to the guide module and the detector module. After the guide features are transmitted to the guide module, the method further includes: the guide module includes two guide layers; the guide module processes the dual-phase image features through the guide features to extract the region of interest in the dual-phase image; and generates an attention feature map corresponding to the region of interest.

[0009] In an exemplary embodiment, after generating the attention feature map corresponding to the region of interest, the method further includes: inputting the attention feature map and the dual-phase image features into the detector module to extract the changed region; updating the changed region in combination with the attention feature map corresponding to the region of interest, retaining the changes in the region of interest, and obtaining the change detection result of the class of interest.

[0010] In an exemplary embodiment, before the initial remote sensing image change detection network is trained using the multimodal remote sensing image change detection dataset, the method further includes: uniformly processing the remote sensing image data in the multimodal remote sensing image change detection dataset and the change labels in the change label range data into an image of a target pixel; and generating a text prompt corresponding to the image of the target pixel.

[0011] In an exemplary embodiment, the inputting of the multimodal prompt into the remote sensing image change detection network to obtain the change detection result of the remote sensing image class of interest includes: inputting a set of the dual-phase images and the text prompt into the remote sensing image change detection network for detection, recognition and segmentation extraction to obtain a binary mask of the class of interest between the dual-phase images; and obtaining the change detection result of the class of interest based on the binary mask.

[0012] According to the second aspect of the present invention, there is also provided a remote sensing image change area detection device based on multimodal prompt guidance, which includes: a construction unit, used to construct a multimodal remote sensing image change detection data set corresponding to the change detection field, wherein the multimodal remote sensing image change detection data set includes remote sensing image data, change label range data and semantic category data of the change area; an establishment unit, used to establish an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture; a training unit, used to train the initial remote sensing image change detection network through the multimodal remote sensing image change detection data set, and determine the remote sensing image change detection network based on the acquired model parameters; a detection unit, used to input the multimodal prompt into the remote sensing image change detection network, and obtain the change detection result of the remote sensing image class of interest.

[0013] According to the third aspect of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned remote sensing image change area detection method based on multimodal prompt guidance when running.

[0014] According to a fourth aspect of the present invention, there is also provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the remote sensing image change area detection method based on multimodal prompt guidance through the computer program.

[0015] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0016] (1) The present invention provides a remote sensing image change region detection method based on multimodal prompt guidance. By constructing a large-scale text and image multimodal remote sensing image change detection dataset, a remote sensing image change detection network based on multimodal prompt guidance is built; the network model is trained to obtain model parameters, and multimodal prompts are used to perform change detection of specific classes of interest. The method adopts a detector-guide architecture, in which the guide generates attention features of the region of interest based on the multimodal prompts input by the user, thereby guiding the detector's attention to focus on the category area of ​​interest to the user. The detector is responsible for extracting all potential change regions in the dual-temporal remote sensing image, and updating the features in combination with the guidance information provided by the guide, and finally outputting the change detection binary result. This method makes full use of the multimodal large model technology, without the need for additional data collection or model fine-tuning, and only needs to input the multimodal prompts and dual-temporal remote sensing images of interest to generate a binary mask of the change target of interest, which effectively reduces the interference of complex change types and realizes accurate detection of change regions of interest.

[0017] (2) The remote sensing image change area detection method based on multimodal prompt guidance provided in this application is adopted. Compared with the existing change detection method, the present invention not only ensures the detection accuracy in multiple scenarios, but also eliminates the need for users to collect data for feature alignment and fine-tuning. It has the advantages of being ready to use, highly automated, and having stable detection effects, which can significantly improve production efficiency and reduce time costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic flow chart of an optional method for detecting a changed area in a remote sensing image based on multimodal prompt guidance provided in an embodiment of the present application;

[0020] Figure 2A schematic flow chart of another optional remote sensing image change area detection method based on multimodal prompt guidance provided in an embodiment of the present application;

[0021] Figure 3 A network overall architecture diagram of an optional remote sensing image change area detection method based on multi-modal prompt guidance provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of an optional guide provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of an optional detector provided in an embodiment of the present application;

[0024] Figure 6 An optional data set provided for an embodiment of the present application involves a type quantity statistical graph;

[0025] Figure 7 A schematic diagram of an optional change detection effect provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of another optional change detection effect provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of the structure of an optional remote sensing image change area detection device based on multi-modal prompt guidance provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0030] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0031] According to one aspect of the embodiments of the present application, a method for detecting a remote sensing image change region based on multimodal prompt guidance is provided. Figure 1 The present invention describes a remote sensing image change area detection method based on multimodal prompt guidance provided in an embodiment of the present application.

[0032] Figure 1 is a flow chart of an optional method for detecting a remote sensing image change region based on multimodal prompt guidance provided in an embodiment of the present application, such as Figure 1 As shown, the process of the method may include the following steps:

[0033] S102, constructing a multimodal remote sensing image change detection dataset corresponding to the change detection field, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data, and semantic category data of the change area;

[0034] S104, establishing an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture;

[0035] S106, training the initial remote sensing image change detection network using the multimodal remote sensing image change detection data set, and determining the remote sensing image change detection network based on the obtained model parameters;

[0036] S108, inputting the multimodal prompt into the remote sensing image change detection network to obtain the change detection result of the class of interest of the remote sensing image.

[0037] The remote sensing image change area detection method based on multimodal prompt guidance provided in this application can be applied in the fields of ecological environment protection, land and resources monitoring, military reconnaissance, etc. By using remote sensing images of the same location at different times, the state differences of surface entities or phenomena in different periods are identified based on image processing and mathematical models. Compared with other remote sensing image interpretation technologies (such as classification and recognition), remote sensing change detection requires images from at least two periods, and the process is more complicated, including high-precision geometric alignment between images, radiation and atmospheric correction, phenological relative normalization, change feature extraction, change area and type determination, accuracy verification and other steps. In order to cope with different remote sensing data characteristics and task requirements, a variety of change detection methods have been developed, such as those based on algebraic calculations, image transformations, and post-classification comparisons.

[0038] Figure 2 A flowchart of another optional remote sensing image change area detection method based on multimodal prompt guidance provided in an embodiment of the present application, combined with Figure 1 and Figure 2As shown, the embodiment of the present application proposes a zero-sample remote sensing image change area detection method based on multimodal prompt guidance, which aims to solve the problem that the existing change detection model is difficult to quickly and directly apply to actual scenes. By constructing a large-scale text and image multimodal remote sensing image change detection dataset; building a remote sensing image change detection network based on multimodal prompt guidance; training the network model and obtaining the model parameters; using multimodal prompts to detect changes in the remote sensing image of interest, the change detection is focused on the area of ​​interest corresponding to the prompt text, so as to achieve stable and universal remote sensing image change detection.

[0039] The remote sensing image change detection network in this embodiment is a large multimodal model. Due to its huge number of parameters, it requires rich data support to avoid overfitting and achieve effective fusion of cross-modal information, thereby enhancing the generalization ability, diversity and robustness of the model. However, the current change detection research field lacks large-scale data sets that can effectively support the training of large models, and the existing change detection data sets are organized in different ways and are difficult to apply directly. To solve this problem, the present invention adopts a unified data organization method that is specifically suitable for the training of multimodal models, and integrates all publicly available change detection data sets in the industry to construct a large-scale graphic multimodal remote sensing image change detection data set.

[0040] The specific integration method is to first collect all common change detection data in the field of change detection, screen out data sets that have dual-temporal remote sensing images, change range labels, and semantic categories of change areas, and then uniformly process them into binary images according to the original change label format type of the data set. The change label format type can be data for single-type change detection, data for multi-type changes but only single-temporal labels, and data for multi-type changes and dual-temporal labels.

[0041] Through the above steps S102 to S108, a multimodal remote sensing image change detection dataset corresponding to the change detection field is constructed, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data and semantic category data of the change area; an initial remote sensing image change detection network guided by multimodal prompts is established, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture; the initial remote sensing image change detection network is trained through the multimodal remote sensing image change detection dataset, and the remote sensing image change detection network is determined based on the acquired model parameters; the multimodal prompts are input into the remote sensing image change detection network to obtain the change detection results of the remote sensing image class of interest, and the binary mask of the change target of interest can be generated by only inputting the multimodal prompts and dual-phase remote sensing images of interest to the user, which effectively reduces the interference of complex change types and realizes accurate detection of change areas of interest.

[0042] In an exemplary embodiment, constructing a multimodal remote sensing image change detection dataset corresponding to the change detection field includes:

[0043] S11, determining the collected change detection data set including remote sensing image data, change label range data and semantic category data of the change area as a multimodal remote sensing image change detection data set;

[0044] S12, processing the remote sensing image data into a binary image according to the change label format type corresponding to the multimodal remote sensing image change detection dataset.

[0045] In this embodiment, after the multimodal remote sensing image change detection dataset is determined, the remote sensing image data may be processed into a binary image according to the change label format type corresponding to the multimodal remote sensing image change detection dataset.

[0046] Optionally, in the case of data with a change label format type of single-type change detection, that is, the data set only contains a binary label of the changes between the previous and next phases, such as the large-scale remote sensing building change detection data set (LEVIR: LEarning, VIsion and Remote sensing laboratory - Change Detection, LEVIR-CD), the building change detection benchmark data set (WuHan University Change Detection, WHU-CD. There are only two pixel values ​​of the label image, such as {0, 1} or {0, 255}, which are directly and uniformly binarized into {0, 255} labels.

[0047] In the case of data with multiple types of changes but only single-phase labels, that is, there is only one labeled image in the dataset, and there are multiple label pixel values, corresponding to multiple semantic categories that have changed. For example, in the Chinese Multi-Class Change Detection Dataset (CNAM-CD), the label of each set of data has only one multi-pixel value image, and each pixel value corresponds to one type of change, for example:

[0048] {Pixel value "0": water body change; pixel value "1": bare land change; pixel value "2": vegetation change}

[0049] For this, all combinations of types can be combined, e.g.

[0050] {Pixel value "0": only water changes;

[0051] Pixel value "1": only the bare ground changes;

[0052] Pixel value “2”: vegetation changes only;

[0053] Pixel values ​​“0” and “1”: water bodies and bare land changes;

[0054] Pixel values ​​“0” and “2”: water bodies and vegetation changes;

[0055] Pixel values ​​“1” and “2”: bare ground and vegetation changes;

[0056] Pixel values ​​"0", "1", and "2": water bodies, bare land, and vegetation changes}

[0057] Each combination change type listed above is processed into a binary {0, 255} label. This multi-type combination labeling method is more difficult to identify than a single type of change label, which helps the model better distinguish the features of each type and learn deeper and more valuable information.

[0058] In the case of data with a change label format type of multi-type change and dual-temporal labels, the dataset change type contains many types of changes, each of which is represented by a pixel value, for example:

[0059] {Pixel value "0": water body; pixel value "1": bare land; pixel value "2": vegetation}

[0060] It should be noted that this type of data contains two labeled images of the previous and next phases (labeled as label A and label B). For this type of data, the positions where the pixel values ​​in the two labeled images change can be extracted, and the change type of each pixel value and its corresponding position can be extracted, for example:

[0061] {Pixel value "0" in A but "1" in B: water body changes to bare land;

[0062] Pixel value "0" in A but "2" in B: water changes to vegetation;

[0063] Pixel value "1" in A but "2" in B: bare land changes to vegetation}

[0064] These change combinations are processed as binary labels of {0, 255} respectively.

[0065] Through this embodiment, a unified data organization method is adopted, which is specifically suitable for the training of multimodal models, and integrates all public change detection data sets in the industry to achieve effective fusion of cross-modal information and enhance the generalization ability, diversity and robustness of the model.

[0066] In an exemplary embodiment, establishing an initial remote sensing image change detection network based on multimodal cue guidance includes:

[0067] S21, the initial remote sensing image change detection network includes a feature extraction module, a guide module and a detector module;

[0068] S22, the feature extraction module is composed of an image feature extraction module and a text feature extraction module, wherein the image feature extraction module in the feature extraction module is used to extract features of the input dual-phase image and encode them into dual-phase image features, and the text feature extraction module in the feature extraction module is used to extract features of the input semantic guidance and encode them into guidance features;

[0069] S23, the dual-phase image features are respectively transmitted to the guide module and the detector module, and the guide features are transmitted to the guide module.

[0070] In this embodiment, Figure 3 The overall network architecture diagram of an optional remote sensing image change area detection method based on multi-modal prompt guidance provided in the embodiment of the present application is as follows: Figure 3 As shown in Figure 1, the initial remote sensing image change detection network mainly consists of a feature extraction module, a guide module based on Transformer (a deep learning model based on self-attention mechanism), and a detector module based on pyramid scene parsing. When the network is running, the user inputs a pair of dual-phase images to be detected. and , and a semantic guide . These three are first input into the feature extraction module for feature extraction and encoded into image features and guidance features. The image features are respectively passed into the guide and the detector, while the guidance features are passed into the guide. The guide uses the guidance features to process the dual-phase image features, extract the region of interest, and generate an attention feature map of the region of interest. The attention feature map is sent to the detector together with the dual-phase image features. The detector extracts the changed region based on the dual-phase region features, and updates the changed region in combination with the attention features of the region of interest, retaining only the changes in the region of interest, thereby achieving the change detection effect of prompt guidance.

[0071] The feature extraction module consists of two parts: the image feature extraction module and the text feature extraction module. For the image encoder, the image encoder can be any network with an output of C×H×W, where C (Channels) is the number of channels; H (Height) is the height. In image data, the height refers to the number of pixels in the image in the vertical direction; W (Width) is the width. In image data, the width refers to the number of pixels in the image in the horizontal direction. In order to achieve scalability and make full use of powerful pre-trained models, a vision model (ViT) based on the Transformer architecture pre-trained by Masked Autoencoders (MAE) can be selected, and minimal adaptive modifications are made to handle high-resolution inputs. Specifically, ViT-H / 16 can be used, which contains a 14×14 window attention and four equidistant global attention blocks. The output of the image encoder is the embedding of the input image after 8 times downsampling. The training image input resolution is 512×512, which is obtained by resizing the image and padding the shorter side, so the image embedding size is 64×64. In order to reduce the channel dimension, 1×1 convolution can be used to reduce the number of channels to 256, and 3×3 convolution is used, also with 256 channels, and each convolution operation is followed by layer normalization. , R represents the dimension of the real number space. The image feature extraction process can be described by the following formula, where: is the number of channels after encoding, Representation layer normalization, Represents the image features after ViT encoding, Represents the final output image embedding:

[0072]

[0073]

[0074] For the text encoder, the BERT model is used to generate text embeddings. BERT (Bidirectional Encoder Representations from Transformers) is a bidirectional self-attention model that is pre-trained through a masked language model and the next sentence prediction task, and can understand the meaning of words in context. Specifically, the BERT-base model can be used, which contains a 12-layer Transformer encoder, each layer contains 12 self-attention heads, the hidden layer size is 768, and the total number of parameters is approximately 110 million. The text encoder first tokenizes the input text and encodes each word through word embedding and position embedding. The maximum sequence length is set to 512 tokens. These embeddings are processed using a multi-layer self-attention mechanism to capture the contextual information in the text, and then the output is average pooled to reduce the embedding dimension to 256. Suppose the input text prompt is , the length after word segmentation is , Represents the BERT encoded features, The final output text prompt embedding is represented by the following formula:

[0075]

[0076]

[0077] Through this embodiment, the change detection network based on the detector-director architecture enables users to obtain change detection results of interest by inputting prompts of arbitrary semantic orientation.

[0078] In an exemplary embodiment, after the dual-phase image features are respectively transmitted to the guide module and the detector module, and the guide features are transmitted to the guide module, the method further includes:

[0079] The guide module includes two guide layers;

[0080] S31, the guide module processes the dual-phase image features through the guide features to extract the region of interest in the dual-phase image;

[0081] S32, generating an attention feature map corresponding to the region of interest.

[0082] In this embodiment, Figure 4 A schematic diagram of the structure of an optional guide provided in an embodiment of the present application, such as Figure 4As shown in Figure 1, the Transformer segmentation model is referenced and the standard Transformer decoder is modified. In order to combine the prompt information with the image embedding, a learned output tag embedding can be inserted into the prompt embedding set for the output of the guide. The guide module can efficiently map the dual-temporal remote sensing image embedding and the prompt embedding to the output mask, that is, the Region of Interest Attention (ROI attention).

[0083] Optionally, you can set the input text prompt mark to , the image is embedded as ,in To calculate the length of the text after tokenization, each guidance layer performs four steps:

[0084] Perform self-attention on the cue marker:

[0085] Cross-attention operation from hint token (as query) to image embedding:

[0086] Update each cue tag using a point-wise MultiLayer Perceptron (MLP)

[0087] Cross-attention operation from image embedding (as query) to hint tokens.

[0088] Through the above steps, the guide is able to effectively fuse the prompt information with the image embedding and extract the mask of the region of interest. In particular, during the cross-attention process, the image embedding is regarded as a set of 64×64×256 dimensional vectors. Each self-attention, cross-attention, and MLP operation is equipped with residual connections, layer normalization, and a dropout of 0.1 during training (inactivation at any time). Each guide layer outputs updated prompt tags and image embeddings, which are passed to the next layer for further processing. Two layers of guide layers are used in this embodiment. In order to ensure that the guide can make full use of geometric information, position encoding is added to the image embedding when the image embedding participates in the attention layer.

[0089] After the guidance layer is run, the updated image embedding is upsampled by a factor of 4 (ultimately downsampled by a factor of 2 relative to the input image) through two 2×2 transposed convolutional layers with a stride of 2. The upsampled image embedding is then again attended to using the hint tokens, and the updated output token embedding is passed to a small three-layer MLP. The MLP outputs a vector that is similar to the upsampled image ( ) is embedded into a vector matching the channel dimension by ) is multiplied with the upsampled image embedding to obtain the mask prediction for a single image Finally, the masks are unified to the original image resolution by interpolation. ) Calculate the maximum value of the pixel position and get the final mask prediction as the attention feature of the region of interest , the process can be expressed as follows:

[0090]

[0091] The Transformer of the guide uses an embedding dimension of 256. The internal dimension of the MLP block is larger, 2048, but it is only applied to a small number of prompt tokens (usually no more than 20), so it does not incur a high computational burden. To further improve computational efficiency, the channel dimensions of the query, key, and value are halved to 128 in the cross-attention layer. When the image embedding size is 64×64, this channel compression can effectively reduce computational overhead. All attention layers use 8 attention heads to ensure the stability and expressiveness of the model under the multi-head attention mechanism. The transposed convolution used to upsample the output image embedding uses the Gaussian Error Linear Unit (GELU) activation function, and each layer of convolution is separated by layer normalization.

[0092] In an exemplary embodiment, after generating the attention feature map corresponding to the region of interest, the method further includes:

[0093] S41, inputting the attention feature map and the dual-phase image features into the detector module to extract the change area;

[0094] S42, updating the changed region in combination with the attention feature map corresponding to the region of interest, retaining the changes in the region of interest, and obtaining the change detection result of the class of interest.

[0095] In this embodiment, Figure 5 A schematic diagram of the structure of an optional detector provided in an embodiment of the present application, such as Figure 5 As shown, the detector module is used to efficiently detect all potential change areas in the dual-phase image, and screen the change areas in combination with the input guided attention feature map, and output the change detection result consistent with the original image resolution.

[0096] Optionally, the input dual-phase images are encoded, and their features are subtracted and their absolute values ​​are calculated to obtain a change feature map. This step captures all the difference information between the dual-phase images, that is, , if the features of the dual-phase image after encoding are recorded as and , the process can be described as:

[0097]

[0098] After encoding, the change features can be sent to the Pyramid Scene Parsing (PSP) segmentation module for further processing. The PSP module introduces the Pyramid Pooling Module, which divides the feature map into different numbers of sub-regions through multiple pooling operations of different scales (such as 1×1, 2×2, 3×3 and 6×6), integrating global and local context information to cope with different size changes of objects and scenes. The pooled features are reduced in dimension through 1×1 convolution and restored to the size of the original feature map through upsampling. Subsequently, these features are concatenated and fused with the original change feature map. The fused features are processed by the convolution layer, and the change detection results that match the input image size are generated through the full convolution layer. Assume the pooling scale is , Indicates the difference information conduct Feature map after scale pooling, represents the updated features after dimensionality reduction and upsampling, represents the multi-scale fusion result after splicing, represents the change detection result initially extracted by the detector. The process can be expressed as:

[0099]

[0100] Furthermore, the change detection results can be and the guided attention feature map provided by the guide module Multiply point by point, as shown in the following formula, this operation updates the change feature map according to the guidance information, retaining only the change features of the area of ​​interest , thereby filtering the areas of interest to users

[0101]

[0102] Through this embodiment, the detector module can use multi-scale context information and the guidance of the region of interest to achieve fine detection of the changed area and generate high-precision change detection results.

[0103] In an exemplary embodiment, before training the initial remote sensing image change detection network using the multimodal remote sensing image change detection dataset, the method further includes:

[0104] S51, uniformly processing the remote sensing image data in the multimodal remote sensing image change detection dataset and the change labels in the change label range data into an image of the target pixel;

[0105] S52, generating a text prompt corresponding to the image of the target pixel.

[0106] In this embodiment, before training the initial remote sensing image change detection network through the multimodal remote sensing image change detection dataset, it is necessary to collect and preprocess the remote sensing image change detection data. Optionally, the embodiment of this application collects 17 public remote sensing image change detection datasets. In order to meet the network input requirements and the limitation of graphics processing unit memory (GPU) capacity, the image and change label can be uniformly processed into a 512×512 pixel image, and the corresponding text description is generated.

[0107] For example, the number of each type of statistics is as follows Figure 6 As shown in the figure, the final total number of samples is 143,452 groups of variation instances, covering common types of objects including buildings, farmlands, man-made objects, forests, wetlands, water, roads, lakes, mountains, trees, shrubs, vegetation, playgrounds, bridges and grasslands. The dataset is divided into training set, validation set and test set in a ratio of 8:1:1, containing 114,761, 14,345 and 14,346 images respectively.

[0108] After preprocessing the data set, the network model can be trained. The model training can be performed using the prepared labeled training data under the Pytorch framework (Python's deep learning library), and the network model parameters with the best performance on the validation set are saved. During the training process, cross-entropy loss (CE Loss) and Dice Loss are used, the total iteration Epoch is set to 200, the optimizer uses the stochastic gradient descent (SGD) algorithm with an initial learning rate of 0.01, the momentum and weight decay coefficients are set to 0.9 and 0.0001 respectively, and the model parameters are initialized using a normal distribution. Then, according to the various processed change labels, the corresponding text prompt labels are made. Considering that the model constructed by the present invention lacks a pre-trained model reference in the field and needs to be retrained, the quality and consistency of the training data are required to be higher. Therefore, the present invention adopts a text template to make text labels, and fills in the corresponding type changes in the text template according to the binary label. The data generated in this way has a consistent format and structure, which helps the model better understand the patterns and characteristics of the input data, and helps the model better understand the relationship between input and output, thereby improving the model's learning efficiency and prediction accuracy. The following are three types of examples:

[0109] The text template is “Identify changes in {type of changes} in the image.”

[0110] For single-type change detection data, you can directly fill in the single category, such as the building change detection dataset, the text prompt label is "Identify changes in buildings in the image."

[0111] For data with multiple types of changes but only a single phase label, it is necessary to fill the corresponding type text into the template according to the combination of the above label extraction. For example, the corresponding relationship between pixel value and type is:

[0112] {Pixel value "0": water body change; pixel value "1": bare land change; pixel value "2": vegetation change}

[0113] There are seven extraction combinations as described above, namely:

[0114] {Pixel value "0": only water changes;

[0115] Pixel value "1": only the bare ground changes;

[0116] Pixel value “2”: vegetation changes only;

[0117] Pixel values ​​“0” and “1”: water bodies and bare land changes;

[0118] Pixel values ​​“0” and “2”: water bodies and vegetation changes;

[0119] Pixel values ​​“1” and “2”: bare ground and vegetation changes;

[0120] Pixel values ​​"0", "1", and "2": water bodies, bare land, and vegetation changes}

[0121] Then, the corresponding text description can be produced as follows:

[0122] {Pixel value "0": Identify changes in water bodies in the image;

[0123] Pixel value "1": Identify changes in bare land in the image;

[0124] Pixel value "2": Identify changes in vegetation in the image;

[0125] Pixel values ​​"0" and "1": Identify changes in water bodies and bare land in the image;

[0126] Pixel values ​​"0" and "2": Identify changes in water bodies and vegetation in the image;

[0127] Pixel values ​​"1" and "2": Identify changes in bare land and vegetation in the image;

[0128] Pixel values ​​"0", "1", and "2": Identify changes in water bodies, bare land, and vegetation in the image}

[0129] For data with multiple types of changes and dual-phase labels, it is necessary to change the type according to the extracted changes before and after, and fill the template with the corresponding change type text. For example, the corresponding relationship between pixel value and type is:

[0130] {Pixel value "0": water body change; pixel value "1": bare land change; pixel value "2": vegetation change}

[0131] As described in the label extraction section above, there are three types of variation extraction:

[0132] {Pixel value "0" in A but "1" in B: water body changes to bare land;

[0133] Pixel value "0" in A but "2" in B: water changes to vegetation;

[0134] Pixel value "1" in A but "2" in B: bare land changes to vegetation}

[0135] Correspondingly, the above changes can be filled in the text template:

[0136] {Pixel value "0" in A but "1" in B: Identify changes in water bodies to bare land in the image;

[0137] Pixel value "0" in A but "2" in B: Identify changes in water to vegetation in the image;

[0138] Pixel value "1" in A but "2" in B: Identify changes in bare land to vegetation in the image}

[0139] Furthermore, all processed dual-temporal remote sensing images and change labels of the data set can be processed into images with a resolution of 512*512: for images and labels with an original resolution greater than 512*512, the corresponding positions are cropped; for images and labels with an original resolution less than 512*512, interpolation is used to resample them to a resolution of 512*512, the image interpolation mode is bilinear interpolation, and the label interpolation mode is nearest neighbor interpolation.

[0140] In an exemplary embodiment, inputting the multimodal prompts into a remote sensing image change detection network, and obtaining the change detection results of the class of interest of the remote sensing image includes:

[0141] S61, inputting a set of dual-temporal images and text prompts into a remote sensing image change detection network for detection, recognition, segmentation and extraction, and obtaining a binary mask of the class of interest between the dual-temporal images;

[0142] S62, obtaining a change detection result of the class of interest based on the binary mask.

[0143] In this embodiment, the trained network model can be used to perform prompt-guided detection, recognition and segmentation extraction of change targets in dual-temporal remote sensing image test data. For example, a pair of dual-temporal remote sensing images and a text prompt "Identify changes in {search type} in the image." can be input to obtain a binary mask of the change target of interest between the dual-temporal images, so as to realize the automated interpretation of the changes of the objects of interest in the dual-temporal remote sensing images.

[0144] The embodiments of the present application conducted zero-sample change detection experiments on the WuHan University Change Detection, WHU-CD and the Global Very-high-resolution Landslide Mapping dataset, GVLM. The experiments only input dual-temporal remote sensing images and text prompts "Identify changes in buildings in the image." and "Identify changes in landslides in the image.", respectively. The model was not fine-tuned. Only with text guidance, the remote sensing image change detection network model maintained a high detection accuracy in a variety of scenarios, demonstrating excellent generalization performance.

[0145] Combination Figure 7 and Figure 8 As shown, Figure 7 are a set of dual-temporal remote sensing images, corresponding binary mask labels of the change regions, and the prediction results of the method of the present invention, respectively. Figure 8 Another set of dual-phase remote sensing images, corresponding binary mask labels of the change regions, and the prediction results of the method of the present invention are shown in Figure 1. From the change detection results, the method of the present invention can accurately identify and locate the change target of interest between the dual-phases according to the guidance prompts input by the user and output its binary mask.

[0146] According to another aspect of the embodiments of the present application, a remote sensing image change area detection device for implementing the above-mentioned remote sensing image change area detection method based on multimodal prompt guidance is also provided. Fig. 9is a schematic structural diagram of an optional remote sensing image change area detection device based on multi-modal prompt guidance according to an embodiment of the present application, such as Fig. 9 As shown, the device may include:

[0147] A construction unit 902 is used to construct a multimodal remote sensing image change detection dataset corresponding to the change detection field, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data, and semantic category data of the change area;

[0148] An establishing unit 904 is used to establish an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture;

[0149] A training unit 906 is used to train the initial remote sensing image change detection network using the multimodal remote sensing image change detection data set, and determine the remote sensing image change detection network based on the acquired model parameters;

[0150] The detection unit 908 is used to input the multimodal prompts into the remote sensing image change detection network to obtain the change detection results of the class of interest of the remote sensing image.

[0151] It should be noted that the construction unit 902 in this embodiment can be used to execute the above step S102, the establishment unit 904 in this embodiment can be used to execute the above step S104, the training unit 906 in this embodiment can be used to execute the above step S106, and the detection unit 908 in this embodiment can be used to execute the above step S108.

[0152] Through the above modules, a multimodal remote sensing image change detection dataset corresponding to the change detection field is constructed, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data and semantic category data of the change area; an initial remote sensing image change detection network guided by multimodal prompts is established, wherein the initial remote sensing image change detection network is a change detection network established based on the detector-guide architecture; the initial remote sensing image change detection network is trained through the multimodal remote sensing image change detection dataset, and the remote sensing image change detection network is determined based on the obtained model parameters; the multimodal prompts are input into the remote sensing image change detection network to obtain the change detection results of the remote sensing image class of interest, and the binary mask of the change target of interest can be generated by only inputting the multimodal prompts and dual-phase remote sensing images of interest to the user, which effectively reduces the interference of complex change types and realizes accurate detection of change areas of interest.

[0153] In an exemplary embodiment, the building block comprises:

[0154] A determination module, configured to determine the acquired change detection data set including the remote sensing image data, the change label range data, and the semantic category data of the change area as the multimodal remote sensing image change detection data set;

[0155] The processing module is used to process the remote sensing image data into a binary image according to the change label format type corresponding to the multimodal remote sensing image change detection data set.

[0156] In an exemplary embodiment, the establishing unit comprises:

[0157] The initial remote sensing image change detection network includes a feature extraction module, a guide module and a detector module;

[0158] The feature extraction module is composed of an image feature extraction module and a text feature extraction module. The image feature extraction module in the feature extraction module is used to extract features from an input dual-phase image and encode them into dual-phase image features. The text feature extraction module in the feature extraction module is used to extract features from an input semantic guide and encode them into a guide feature.

[0159] The dual-phase image features are respectively transmitted to the guide module and the detector module, and the guide features are transmitted to the guide module.

[0160] In an exemplary embodiment, the apparatus further comprises:

[0161] The guide module includes two guide layers;

[0162] A first extraction unit, configured for the guide module to process the dual-phase image feature through the guide feature to extract a region of interest in the dual-phase image;

[0163] The first generating unit is used to generate an attention feature map corresponding to the region of interest.

[0164] In an exemplary embodiment, the apparatus further comprises:

[0165] A second extraction unit, configured to input the attention feature map and the dual-phase image features into the detector module to extract a change region;

[0166] An updating unit is used to update the changed area in combination with the attention feature map corresponding to the area of ​​interest, retain the changes in the area of ​​interest, and obtain the change detection result of the class of interest.

[0167] In an exemplary embodiment, the apparatus further comprises:

[0168] A processing unit, configured to uniformly process the remote sensing image data in the multimodal remote sensing image change detection data set and the change labels in the change label range data into an image of a target pixel;

[0169] The second generating unit is used to generate a text prompt corresponding to the image of the target pixel.

[0170] In an exemplary embodiment, the detection unit comprises:

[0171] An input module, used for inputting a set of the dual-phase images and the text prompt into the remote sensing image change detection network for detection, recognition, segmentation and extraction, and obtaining a binary mask of the class of interest between the dual-phase images;

[0172] An acquisition module is used to obtain the change detection result of the class of interest based on the binary mask.

[0173] It should be noted here that the examples and scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned modules as part of the device can run in a hardware environment and can be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0174] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above-mentioned methods for detecting remote sensing image change areas based on multimodal prompt guidance in the embodiments of the present application.

[0175] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0176] S1, constructing a multimodal remote sensing image change detection dataset corresponding to the change detection field, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data, and semantic category data of the change area;

[0177] S2, establishing an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture;

[0178] S3, training the initial remote sensing image change detection network through the multimodal remote sensing image change detection dataset, and determining the remote sensing image change detection network based on the obtained model parameters;

[0179] S4, input the multimodal prompts into the remote sensing image change detection network to obtain the change detection results of the class of interest of the remote sensing image.

[0180] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0181] Among them, computer-readable storage media may include, but are not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0182] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned remote sensing image change area detection method based on multimodal prompt guidance is also provided. The electronic device can be a server, a terminal, or a combination thereof.

[0183] Fig.10 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Fig.10 As shown, it includes a processor 1002, a communication interface 1004, a memory 1006 and a communication bus 1008, wherein the processor 1002, the communication interface 1004, and the memory 1006 communicate with each other through the communication bus 1008, wherein:

[0184] Memory 1006, used for storing computer programs;

[0185] The processor 1002 is used to execute the computer program stored in the memory 1006 to implement the following steps:

[0186] S1, constructing a multimodal remote sensing image change detection dataset corresponding to the change detection field, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data, and semantic category data of the change area;

[0187] S2, establishing an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture;

[0188] S3, training the initial remote sensing image change detection network through the multimodal remote sensing image change detection dataset, and determining the remote sensing image change detection network based on the obtained model parameters;

[0189] S4, input the multimodal prompts into the remote sensing image change detection network to obtain the change detection results of the class of interest of the remote sensing image.

[0190] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The communication interface is used for communication between the electronic device and other devices.

[0191] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0192] As an example, the memory 1006 may include, but is not limited to, the construction unit 902, the establishment unit 904, the training unit 906, and the detection unit 908 in the remote sensing image change area detection device based on multimodal prompt guidance. In addition, other module units in the remote sensing image change area detection device based on multimodal prompt guidance may also be included but are not limited to, which will not be repeated in this example.

[0193] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0194] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0195] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0196] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0198] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0201] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0202] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0203] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0204] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image change region detection method based on multimodal prompt guidance, characterized in that: include: Constructing a multimodal remote sensing image change detection dataset corresponding to the change detection field, wherein the multimodal remote sensing image change detection dataset includes remote sensing image data, change label range data, and semantic category data of the change area; Establishing an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture; The initial remote sensing image change detection network is trained using the multimodal remote sensing image change detection data set, and the remote sensing image change detection network is determined based on the obtained model parameters; Inputting the multimodal prompts into the remote sensing image change detection network to obtain change detection results of the class of interest of the remote sensing image; The establishment of an initial remote sensing image change detection network based on multimodal prompt guidance includes: The initial remote sensing image change detection network includes a feature extraction module, a guide module and a detector module; The feature extraction module is composed of an image feature extraction module and a text feature extraction module. The image feature extraction module in the feature extraction module is used to extract features from an input dual-phase image and encode them into dual-phase image features. The text feature extraction module in the feature extraction module is used to extract features from an input semantic guide and encode them into a guide feature. The dual-phase image features are respectively transmitted to the guide module and the detector module, and the guide features are transmitted to the guide module; After the dual-phase image features are respectively transmitted to the guide module and the detector module, and the guide features are transmitted to the guide module, the method further includes: The guide module includes two guide layers; The guide module processes the dual-phase image features through the guide features to extract the region of interest in the dual-phase image; generating an attention feature map corresponding to the region of interest; After generating the attention feature map corresponding to the region of interest, the method further includes: Inputting the attention feature map and the dual-phase image features into the detector module to extract the changed area; The changed region is updated in combination with the attention feature map corresponding to the region of interest, the changes in the region of interest are retained, and the change detection result of the class of interest is obtained.

2. The method for detecting changed areas of remote sensing images based on multimodal prompt guidance according to claim 1, characterized in that: The construction of a multimodal remote sensing image change detection dataset corresponding to the change detection field includes: Determining the collected change detection data set including the remote sensing image data, the change label range data, and the semantic category data of the change area as the multimodal remote sensing image change detection data set; The remote sensing image data is processed into a binary image according to the change label format type corresponding to the multimodal remote sensing image change detection data set.

3. The remote sensing image change area detection method based on multi-modal prompt guidance as claimed in claim 1, characterized in that: Before training the initial remote sensing image change detection network using the multimodal remote sensing image change detection dataset, the method further includes: The remote sensing image data in the multimodal remote sensing image change detection dataset and the change labels in the change label range data are uniformly processed into an image of a target pixel; A textual hint is generated corresponding to the image of the target pixel.

4. The method for detecting changed areas in remote sensing images based on multimodal prompt guidance according to any one of claims 1 to 3, characterized in that: The step of inputting the multimodal prompts into the remote sensing image change detection network to obtain the change detection results of the class of interest of the remote sensing image includes: Input a set of dual-temporal images and text prompts into a remote sensing image change detection network for detection, recognition, and segmentation extraction, and obtain a binary mask of the class of interest between the dual-temporal images; A change detection result of the class of interest is obtained based on the binary mask.

5. A remote sensing image change area detection device based on multi-modal prompt guidance, which executes the remote sensing image change area detection method based on multi-modal prompt guidance as claimed in claim 1, characterized in that: include: A construction unit is used to construct a multimodal remote sensing image change detection data set corresponding to the change detection field, wherein the multimodal remote sensing image change detection data set includes remote sensing image data, change label range data, and semantic category data of the change area; An establishing unit, used for establishing an initial remote sensing image change detection network based on multimodal prompt guidance, wherein the initial remote sensing image change detection network is a change detection network established based on a detector-guide architecture; A training unit, configured to train the initial remote sensing image change detection network using the multimodal remote sensing image change detection data set, and determine the remote sensing image change detection network based on the acquired model parameters; The detection unit is used to input the multimodal prompts into the remote sensing image change detection network to obtain the change detection results of the class of interest of the remote sensing image.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 4 when executed.

7. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 4 through the computer program.

Images