Remote sensing image segmentation method and device based on large language model

Abstract

The invention discloses a remote sensing image segmentation method and device based on a large language model, a storage medium and electronic equipment. The method comprises the following steps: acquiring a remote sensing image and a natural language corresponding to the remote sensing image; performing multi-scale feature extraction and fusion on the remote sensing image to obtain a multi-scale fusion feature map; encoding the natural language to obtain an embedded vector, processing the multi-scale fusion feature map through a visual token compression mechanism to obtain a compressed image feature, and aligning the embedded vector and the compressed image feature to obtain a cross-modal semantic feature; and generating initial masks based on the multi-scale fusion feature map and the cross-modal semantic features, and performing weighted fusion on all the initial masks to obtain a final semantic target mask. According to the method, a remote sensing image segmentation task with high semantic complexity can be realized, and the method has remarkable intellectualization and practicability advantages.

Description

Technical Field

[0001] This application relates to the field of remote sensing image segmentation technology, and in particular to a remote sensing image segmentation method, apparatus, storage medium and electronic device based on a large language model. Background Technology

[0002] Remote sensing images have wide applications in fields such as emergency response, urban management, and military reconnaissance.

[0003] Traditional remote sensing image analysis methods (such as semantic segmentation and object detection) rely on explicit instructions and fixed classification systems, making them unable to handle complex queries that require spatial context reasoning and domain knowledge. For example, in disaster management, target areas need to be inferred based on unstructured instructions such as "areas prone to landslides," involving the analysis of multiple factors such as slope, vegetation cover, and adjacent infrastructure. Furthermore, traditional models mostly rely on local pixel or regional features for classification, lacking the ability to model the semantic relationships between spatial entities in the image, making it difficult to perform complex spatial semantic combinations and reasoning. Summary of the Invention

[0004] This application provides a remote sensing image segmentation method, apparatus, storage medium, and electronic device based on a large language model, which can achieve remote sensing image segmentation tasks with high semantic complexity and has significant advantages in intelligence and practicality.

[0005] This application provides a remote sensing image segmentation method based on a large language model, including: Obtain remote sensing images and their corresponding natural language; Multi-scale feature extraction and fusion are performed on the remote sensing image to obtain a multi-scale fused feature map; The natural language is encoded to obtain an embedding vector. The multi-scale fusion feature map is processed by a visual token compression mechanism to obtain compressed image features. The embedding vector and the compressed image features are aligned to obtain cross-modal semantic features. An initial mask is generated based on the multi-scale fused feature map and the cross-modal semantic features. The final semantic target mask is obtained by weighted fusion of all the initial masks.

[0006] Furthermore, according to the above-mentioned remote sensing image segmentation method based on a large language model, the remote sensing image undergoes multi-scale feature extraction and fusion to obtain a multi-scale fused feature map, including: The remote sensing images are standardized; The standardized remote sensing images are input into the visual model to extract multi-level spatial features, resulting in feature maps at multiple scales. Upsampling and realigning of feature maps at multiple scales are performed to obtain multiple aligned feature maps. Dynamic weights are generated for each scale, and a multi-scale fused feature map is obtained based on the dynamic weights and the corresponding aligned feature map. Furthermore, according to the above-mentioned remote sensing image segmentation method based on a large language model, the natural language is encoded to obtain an embedding vector, the multi-scale fusion feature map is processed through a visual token compression mechanism to obtain compressed image features, and the embedding vector and compressed image features are aligned to obtain cross-modal semantic features, including: The natural language is encoded using a large language model to generate embedding vectors; Structural similarity detection and semantic importance evaluation are performed on the multi-scale fused feature map. Based on the structural similarity detection results and semantic importance evaluation results, tokens are selected for retention. Compressed image features are obtained based on the retained tokens. A multi-layer cross-attention mechanism is employed to establish a fine-grained alignment relationship between the embedded vector and the compressed image features, thereby obtaining cross-modal semantic features.

[0007] Furthermore, according to the above-mentioned remote sensing image segmentation method based on a large language model, an initial mask is generated based on the multi-scale fused feature map and the cross-modal semantic features. The final semantic target mask is obtained by weighted fusion of all the initial masks, including: A global query vector is constructed based on the cross-modal semantic features, and the multi-scale fused feature map is interacted with the global query vector to obtain an initial mask; The final semantic target mask is obtained by weighted fusion of all initial masks based on the semantic response of the initial mask.

[0008] Furthermore, according to the above-mentioned remote sensing image segmentation method based on a large language model, the final semantic target mask is obtained by weighted fusion of all initial masks based on the semantic response of the initial masks, as expressed by the following formula:

[0009] in, For learnable fusion coefficients, for Activation function This is the initial mask.

[0010] Furthermore, according to the above-described remote sensing image segmentation method based on a large language model, the method further includes: The natural language is constructed into a semantic graph, which includes nodes and edges. The entities corresponding to the semantics are constructed as nodes, and the relationship types and directions between entities are constructed as edges. Node features are constructed through word embedding. The semantic graph is input into a graph attention network to update the node features of the semantic graph; The updated semantic map is fused with the multi-scale fusion feature map to obtain cross-modal semantic features.

[0011] Furthermore, according to the above-described remote sensing image segmentation method based on a large language model, the method further includes: The cross-modal semantic features are input into the semantic sub-control decoder to generate multiple semantic query sub-vectors. Each semantic query sub-vector is used as an independent query key and cross-scale attention inference is performed with the multi-scale fused feature map to generate a corresponding response map. The final semantic target mask is generated based on multiple response maps.

[0012] This application also provides a remote sensing image segmentation device based on a large language model, including: The acquisition module acquires remote sensing images and the corresponding natural language. The multi-scale fusion module is used to extract and fuse multi-scale features from the remote sensing image to obtain a multi-scale fused feature map. The cross-modal semantic feature generation module is used to encode the natural language to obtain an embedding vector, process the multi-scale fusion feature map through a visual token compression mechanism to obtain compressed image features, and align the embedding vector and the compressed image features to obtain cross-modal semantic features. The mask generation module is used to generate an initial mask based on the multi-scale fused feature map and the cross-modal semantic features, and to perform weighted fusion on all the initial masks to obtain the final semantic target mask.

[0013] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described remote sensing image segmentation methods based on a large language model.

[0014] This application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the remote sensing image segmentation method based on a large language model described above.

[0015] This application provides a remote sensing image segmentation method, apparatus, storage medium, and electronic device based on a large language model. This application obtains a cross-modal semantic feature map by aligning the embedding vectors obtained from natural language with a compressed multi-scale fusion feature map. A final semantic target mask is generated based on the multi-scale fusion feature map and the cross-modal semantic features. This application can handle remote sensing image reasoning tasks with high semantic complexity, such as "suitable locations for temporary helipads" or "flood evacuation and resettlement areas," and completes the entire process of "language understanding—visual perception—spatial reasoning—pixel mask output" through a unified model, exhibiting significant advantages in intelligence and practicality. Attached Figure Description

[0016] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0017] Figure 1 A flowchart of a remote sensing image segmentation method based on a large language model provided in an embodiment of this application.

[0018] Figure 2 A schematic diagram of the structure of the remote sensing image segmentation device based on a large language model provided in the embodiments of this application.

[0019] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a remote sensing image segmentation method, apparatus, storage medium, and electronic device based on a large language model. The remote sensing image segmentation apparatus based on a large language model provided in this application can be integrated into an electronic device, which can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0022] Please see Figure 1 , Figure 1 The flowchart illustrates a remote sensing image segmentation method based on a large language model, provided in this application embodiment. This method, applied in electronic devices, includes the following steps: S1, obtain the remote sensing image and its corresponding natural language.

[0023] Each remote sensing image is paired with one or more question-and-answer pairs. The questions reflect implicit semantic needs, such as "Where is a suitable place for a rapid helicopter landing in an emergency medical situation?" or "Where should people gather after an earthquake?" The answers can be generated by GPT-4V or written by experts, and semantically correspond to specific areas in the image, such as "hospital rooftop", "office building rooftop", or "sports field".

[0024] S2, performs multi-scale feature extraction and fusion on remote sensing images to obtain a multi-scale fused feature map.

[0025] Remote sensing images often contain ground features with vastly different scales; for example, a single image may contain large-scale squares and stadiums as well as small-scale tents and vehicles. To effectively address this problem of "coexistence of multiple scale features," this application employs the Swin Transformer to extract feature maps at multiple scales.

[0026] In one embodiment, step S2 includes the following steps: S21, standardize the remote sensing images.

[0027] The input remote sensing image is normalized to a size of 1024×1024.

[0028] S22, the standardized remote sensing image is input into the visual model for multi-level spatial feature extraction to obtain feature maps at multiple scales.

[0029] The Swin Transformer is used to extract four levels of spatial feature maps from the image, corresponding to downsampling ratios of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively. These feature layers are complementary. High-resolution layers (1 / 4, 1 / 8) focus more on local details, such as small targets like vehicles and rescue facilities; The low-resolution layers (1 / 16, 1 / 32) provide regional extent structures, such as global semantic layouts of road networks and building cluster boundaries.

[0030] S23, perform upsampling and realignment operations on feature maps at multiple scales to obtain multiple aligned feature maps.

[0031] To further alleviate the problem of semantic representation instability caused by differences in the size of ground features, this invention introduces a cross-scale feature interaction module to establish a dynamic fusion mechanism between multiple feature layers, thereby improving the model's ability to uniformly perceive multi-scale targets.

[0032] Specifically, let the feature maps extracted at the four scales be as follows: Corresponding to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions, and adjusted to the same spatial size through upsampling operations, the aligned feature maps are obtained. .

[0033] S24, generate dynamic weights for each scale, and obtain a multi-scale fused feature map based on the dynamic weights and the corresponding aligned feature maps. A semantic attention weight generation module is introduced to assign dynamic weights to each scale:

[0034] In the formula, For learnable parameters, This indicates global average pooling.

[0035] The multi-scale fused feature map is represented as follows:

[0036] S3 encodes the natural language to obtain the embedding vector, processes the multi-scale fusion feature map through the visual token compression mechanism to obtain the compressed image features, and aligns the embedding vector and the compressed image features to obtain cross-modal semantic features.

[0037] In one embodiment, step S3 includes the following steps: S31 encodes natural language using a large language model to generate embedding vectors.

[0038] First, the natural language is preprocessed and standardized. The standardized natural language is then input into a large language model for positional encoding to obtain an embedding vector. This embedding vector contains contextual semantics and the underlying reasoning logic corresponding to the semantics. It not only includes explicit semantics (such as "playground"), but also has the ability to understand the implicit needs in the sentence structure (such as "can land", "open space", "convenient transportation").

[0039] S32, perform structural similarity detection and semantic importance evaluation on the multi-scale fused feature map, select the tokens to be retained based on the structural similarity detection results and semantic importance evaluation results, and obtain the compressed image features based on the retained tokens.

[0040] To improve the efficiency of visual representation, a visual token compression module is introduced. This module extracts token subsets corresponding to semantically key regions from multi-scale fused feature maps through redundancy suppression and structural similarity analysis, significantly reducing computational cost while preserving key information.

[0041] Specifically, firstly, structural similarity detection is performed on different multi-scale fused feature maps, and pixel-level structural similarity is calculated to obtain structural similarity detection results. Then, semantic importance is evaluated for each multi-scale fused feature map, assigning "information value scores" to different regions to obtain semantic importance evaluation results. Based on the structural similarity detection results and semantic importance evaluation results, a comprehensive score calculation, selective sampling, and information recombination are performed to obtain the retained tokens. The semantic regions corresponding to the retained tokens are the compressed image features.

[0042] S33 employs a multi-layer cross-attention mechanism to establish a fine-grained alignment relationship between the embedded vector and the compressed image features, thereby obtaining cross-modal semantic features.

[0043] A cross-modal fusion module is constructed, employing a multi-layer cross-attention mechanism to establish a fine-grained alignment relationship between the embedded vectors and compressed image features, achieving semantic-level fusion. The fusion result is then encoded through a multi-head attention layer to output unified cross-modal semantic features.

[0044] Specifically, using image features as the query and text features as the key / value pair, an attention weight matrix is ​​calculated to generate visually enhanced features from the text perspective; using text features as the query and image features as the key / value pair, an attention weight matrix is ​​calculated to generate context-aware features from the image perspective; the concatenated and compressed image features and the cross-attention results are processed through a fully connected layer + sigmoid function to generate fusion weights (between 0 and 1), and weighted summation is performed to achieve adaptive fusion, resulting in the fused result. Multi-head self-attention computation is then performed on the fused result to obtain cross-modal semantic features.

[0045] In one embodiment, semantic features and multi-scale fusion feature maps can also be fused to obtain cross-modal semantic features by constructing a graph. Specific steps include: Natural language is constructed as a semantic graph, which includes nodes and edges. The entities corresponding to the semantics are constructed as nodes, and the relationship types and directions between entities are constructed as edges. Node features are constructed through word embedding. The semantic graph is input into the graph attention network to update the node features of the semantic graph; The updated semantic map is fused with the multi-scale fusion feature map to obtain cross-modal semantic features.

[0046] By introducing semantic reasoning structures from language priors, language embeddings are constructed into structured graph forms (such as entity-relation pairs). The semantic relations in the semantic graph are modeled through the graph attention network GAT, enabling the model to have the ability to combine spatial semantics at the structural level.

[0047] S4 generates an initial mask based on multi-scale fused feature maps and cross-modal semantic features, and then performs weighted fusion on all the initial masks to obtain the final semantic target mask.

[0048] In one embodiment, step S4 includes: S41, construct a global query vector based on cross-modal semantic features, and interact with the multi-scale fused feature map and the global query vector to obtain the initial mask.

[0049] Specifically, the cross-modal semantic features are averaged to obtain a global query vector, which is used as the Query. The multi-scale fused feature map is used as the Key and Value. Attention weights are calculated and aggregated to obtain interaction features. The interaction features are then passed through a prediction head (such as 1×1 convolution + Sigmoid) to obtain an initial mask.

[0050] S42, weighted fusion of all initial masks based on the semantic response of the initial masks to obtain the final semantic target mask.

[0051] A weight is assigned to the initial mask to reflect its semantic reliability, and the response is evaluated based on this weight. This is specifically expressed by the following formula:

[0052] in, For learnable fusion coefficients, for Activation function This is the initial mask.

[0053] In one embodiment, multi-target semantic reasoning is also supported. This involves introducing multiple query vectors to reason about possible target areas such as "playground" and "rooftop," and using target differentiation loss to prevent mask overlap and blurring. The final output is obtained by weighted fusion of mask branches from all levels, including the following steps: Cross-modal semantic features are input into the semantic sub-control decoder to generate multiple semantic query sub-vectors. Each semantic query sub-vector is used as an independent query key and cross-scale attention inference is performed with the multi-scale fused feature map to generate the corresponding response map. The final semantic target mask is generated based on multiple response maps.

[0054] Specifically, firstly, the cross-modal semantic features T are fed into the semantic subspace decoder, where the latent semantic subcategories are deconstructed to generate K semantic query subvectors. Each vector represents a latent target intent (e.g., "rooftop", "playground", "grass"). Each sub-vector serves as an independent query key, and is used in cross-scale attention inference with the multi-scale fused feature map to generate a semantically category-specific response map. And by fusing them along the scale dimension, the final semantic target mask is obtained:

[0055] in, The learnable fusion coefficients corresponding to the response graph. For response diagrams, For target mask.

[0056] all It can be output individually or stacked together to generate a joint semantic coverage graph, which can be used to complete one-to-many spatial instruction reasoning tasks.

[0057] To avoid spatial boundary conflicts and fuzzy overlap between multiple target masks, a target differentiation constraint loss is introduced:

[0058] In the formula, This is a pixel-by-pixel multiplication operation. , For different target masks, The target mask represents the number of target masks. This aims to suppress overlapping areas between semantic masks, improving the spatial discriminability and semantic clarity of the output mask.

[0059] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of a remote sensing image segmentation device based on a large language model. The remote sensing image segmentation device based on a large language model can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0060] The above-mentioned remote sensing image segmentation method based on a large language model can be applied to remote sensing image segmentation models. The training process of a remote sensing image segmentation model is briefly introduced below: (1) Constructing the dataset We construct an image-language-mask triplet training set using publicly available remote sensing datasets. The questions express implicit intentions in natural language (e.g., "Where can I set up a tent?"), and the answers are regions in the image that meet the semantic requirements, labeled as pixel-level masks. Semantic answers can be generated by a large language model or labeled with expert assistance. Each image contains multiple question-answer pairs, enhancing the semantic diversity and complexity of the training.

[0061] (2) Input the dataset into the remote sensing image segmentation model to obtain the final semantic target mask (the specific training process and application process of the model are consistent, and you can refer to steps S1-S4).

[0062] (3) Construct a loss function based on the final semantic target mask and the labeled pixel-level mask, and iteratively train the remote sensing image segmentation model based on the loss function.

[0063] Please see Figure 2 , Figure 2 This application provides a specific description of a remote sensing image segmentation device based on a large language model, which is applied in electronic devices. The device may include: The acquisition module acquires remote sensing images and the corresponding natural language. The multi-scale fusion module is used to extract and fuse multi-scale features from the remote sensing image to obtain a multi-scale fused feature map. The cross-modal semantic feature generation module is used to encode the natural language to obtain an embedding vector, process the multi-scale fusion feature map through a visual token compression mechanism to obtain compressed image features, and align the embedding vector and the compressed image features to obtain cross-modal semantic features. The mask generation module is used to generate an initial mask based on the multi-scale fused feature map and the cross-modal semantic features, and to perform weighted fusion on all the initial masks to obtain the final semantic target mask.

[0064] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0065] In addition, this application also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the remote sensing image segmentation method based on a large language model provided in this application. Therefore, it can achieve the beneficial effects that any remote sensing image segmentation method based on a large language model provided in this invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0066] Figure 3 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the remote sensing image segmentation method based on a large language model provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0067] RF circuit 510 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 510 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0068] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, such as taking pictures with the front-facing camera, processing the captured images, and switching the display colors of the content displayed on the screen. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0069] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control. Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0070] Audio circuitry 560, speaker 561, and microphone 562 provide an audio interface between the user and electronic device 500. Audio circuitry 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. Conversely, microphone 562 converts collected sound signals into electrical signals, which are then received by audio circuitry 560, converted back into audio data, and processed by processor 580. The audio data is then transmitted via RF circuitry 510 to, for example, another terminal, or output to memory 520 for further processing. Audio circuitry 560 may also include an earphone jack to facilitate communication between external headphones and electronic device 500.

[0071] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0072] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0073] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0074] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Obtain remote sensing images and their corresponding natural language; Multi-scale feature extraction and fusion are performed on the remote sensing image to obtain a multi-scale fused feature map; The natural language is encoded to obtain an embedding vector. The multi-scale fusion feature map is processed by a visual token compression mechanism to obtain compressed image features. The embedding vector and the compressed image features are aligned to obtain cross-modal semantic features. An initial mask is generated based on the multi-scale fused feature map and the cross-modal semantic features. All initial masks are then weighted and fused to obtain the final semantic target mask. In practice, each of the above modules can be implemented as an independent entity or arbitrarily combined as the same or several entities. For details on the specific implementation of each module, please refer to the preceding method embodiments; further elaboration is not provided here.

[0075] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the remote sensing image segmentation method based on a large language model provided by the present invention.

[0076] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0077] Since the instructions stored in the storage medium can execute the steps in any embodiment of the remote sensing image segmentation method based on a large language model provided in the embodiments of the present invention, the beneficial effects that any remote sensing image segmentation method based on a large language model provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0078] The foregoing has provided a detailed description of a remote sensing image segmentation method, apparatus, storage medium, and electronic device based on a large language model, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A remote sensing image segmentation method based on a large language model, characterized in that, The method includes: Obtain remote sensing images and their corresponding natural language; Multi-scale feature extraction and fusion are performed on the remote sensing image to obtain a multi-scale fused feature map; The natural language is encoded to obtain an embedding vector. The multi-scale fusion feature map is processed by a visual token compression mechanism to obtain compressed image features. The embedding vector and the compressed image features are aligned to obtain cross-modal semantic features. An initial mask is generated based on the multi-scale fused feature map and the cross-modal semantic features. The final semantic target mask is obtained by weighted fusion of all the initial masks.

2. The remote sensing image segmentation method based on a large language model according to claim 1, characterized in that, Multi-scale feature extraction and fusion are performed on the remote sensing image to obtain a multi-scale fused feature map, including: The remote sensing images are standardized; The standardized remote sensing images are input into the visual model to extract multi-level spatial features, resulting in feature maps at multiple scales. Upsampling and realigning of feature maps at multiple scales are performed to obtain multiple aligned feature maps. Dynamic weights are generated for each scale, and a multi-scale fused feature map is obtained based on the dynamic weights and the corresponding aligned feature map.

3. The remote sensing image segmentation method based on a large language model according to claim 1, characterized in that, The natural language is encoded to obtain an embedding vector. The multi-scale fusion feature map is processed using a visual token compression mechanism to obtain compressed image features. The embedding vector and the compressed image features are aligned to obtain cross-modal semantic features, including: The natural language is encoded using a large language model to generate embedding vectors; Structural similarity detection and semantic importance evaluation are performed on the multi-scale fused feature map. Based on the structural similarity detection results and semantic importance evaluation results, tokens are selected for retention. Compressed image features are obtained based on the retained tokens. A multi-layer cross-attention mechanism is employed to establish a fine-grained alignment relationship between the embedded vector and the compressed image features, thereby obtaining cross-modal semantic features.

4. The remote sensing image segmentation method based on a large language model according to claim 1, characterized in that, An initial mask is generated based on the multi-scale fused feature map and the cross-modal semantic features. All initial masks are then weighted and fused to obtain the final semantic target mask, including: A global query vector is constructed based on the cross-modal semantic features, and the multi-scale fused feature map is interacted with the global query vector to obtain an initial mask; The final semantic target mask is obtained by weighted fusion of all initial masks based on the semantic response of the initial mask.

5. The remote sensing image segmentation method based on a large language model according to claim 4, characterized in that, The final semantic target mask is obtained by weighted fusion of all initial masks based on the semantic response of the initial masks, as expressed by the following formula: in, For learnable fusion coefficients, for Activation function This is the initial mask.

6. The remote sensing image segmentation method based on a large language model according to claim 3, characterized in that, The method further includes: The natural language is constructed into a semantic graph, which includes nodes and edges. The entities corresponding to the semantics are constructed as nodes, and the relationship types and directions between entities are constructed as edges. Node features are constructed through word embedding. The semantic graph is input into a graph attention network to update the node features of the semantic graph; The updated semantic map is fused with the multi-scale fusion feature map to obtain cross-modal semantic features.

7. The remote sensing image segmentation method based on a large language model according to claim 4, characterized in that, The method further includes: The cross-modal semantic features are input into the semantic sub-control decoder to generate multiple semantic query sub-vectors. Each semantic query sub-vector is used as an independent query key and cross-scale attention inference is performed with the multi-scale fused feature map to generate a corresponding response map. The final semantic target mask is generated based on multiple response maps.

8. A remote sensing image segmentation device based on a large language model, characterized in that, include: The acquisition module acquires remote sensing images and the corresponding natural language. The multi-scale fusion module is used to extract and fuse multi-scale features from the remote sensing image to obtain a multi-scale fused feature map. The cross-modal semantic feature generation module is used to encode the natural language to obtain an embedding vector, process the multi-scale fusion feature map through a visual token compression mechanism to obtain compressed image features, and align the embedding vector and the compressed image features to obtain cross-modal semantic features. The mask generation module is used to generate an initial mask based on the multi-scale fused feature map and the cross-modal semantic features, and to perform weighted fusion on all the initial masks to obtain the final semantic target mask.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the remote sensing image segmentation method based on a large language model as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The method includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps in the remote sensing image segmentation method based on a large language model as described in any one of claims 1 to 7.

Images