Multi-modal fusion segmentation method and device based on prior information and mamba mixed model

Abstract

Embodiments of the present disclosure provide a multi-modal fusion segmentation method and device based on prior information and Mamba mixed model, which are applied to the technical field of remote sensing image multi-modal fusion segmentation. The method comprises: acquiring an RGB remote sensing image set and a DSM remote sensing image set as a remote sensing image set to be processed; inputting the remote sensing image set to be processed into a pre-trained remote sensing image multi-modal fusion segmentation network; and outputting a ground feature interpretation semantic segmentation result of the remote sensing image set to be processed. In this way, the prior information can be combined while the multi-modal fusion is performed, the RGB and DSM remote sensing images are effectively fused, and thus the semantic segmentation accuracy is improved.

Description

Technical Field

[0001] This disclosure relates to the field of multimodal fusion segmentation technology for remote sensing images, and in particular to a multimodal fusion segmentation method and apparatus based on prior information and a Mamba hybrid model. Background Technology

[0002] Remote sensing image segmentation is a crucial step in interpreting ground features from remote sensing images. This technology utilizes artificial intelligence algorithms to perform end-to-end pixel-by-pixel annotation on preprocessed remote sensing images. Multimodal fusion segmentation of remote sensing images combines information from multiple modalities, mitigating the problem of class inseparability caused by limitations in image quality or small intra-class differences in a single modality (e.g., shrubs and woodland in RGB images). The mainstream methods in multimodal fusion segmentation of remote sensing images are dual-encoder networks based on CNNs or dual-encoder networks based on a fusion of Transformer and CNN. Among these, CNN-based methods are computationally efficient and have strong local modeling capabilities, but lack global modeling capabilities. Methods based on the fusion of Transformer and CNN can effectively address the lack of global modeling, but the Transformer method is limited by self-attention operations, introducing computational costs that grow exponentially with the input sequence. This means that balancing global and local modeling during the fusion process introduces excessive computational overhead. Furthermore, large differences between modalities can easily lead to fusion difficulties or reduced accuracy due to insufficient utilization of modal information after fusion.

[0003] Therefore, there is an urgent need for a multimodal fusion and segmentation method that can maintain a small computational load while overcoming the difficulties in fusion and the decrease in accuracy caused by large differences between modes. Summary of the Invention

[0004] This disclosure provides a multimodal fusion segmentation method and apparatus based on prior information and a Mamba hybrid model.

[0005] According to a first aspect of this disclosure, a multimodal fusion segmentation method based on prior information and a Mamba hybrid model is provided. The method includes:

[0006] Acquire RGB and DSM remote sensing image sets as the remote sensing image sets to be processed;

[0007] The set of remote sensing images to be processed is input into a pre-trained remote sensing image multimodal fusion and segmentation network;

[0008] Output the semantic segmentation results of the ground features in the remote sensing image set to be processed.

[0009] In addition to the aspects and any possible implementations described above, another implementation is provided.

[0010] The remote sensing image multimodal fusion and segmentation network is trained through the following steps:

[0011] Obtain a target remote sensing image set; wherein, the target remote sensing image set is a subset of the remote sensing image set to be processed;

[0012] The target remote sensing image set is labeled to obtain a category label set;

[0013] Edge extraction is performed on the category label set to obtain the edge label set;

[0014] The target remote sensing image set, category label set, and edge label set are divided into a training set and a validation set according to a preset ratio;

[0015] The pre-constructed remote sensing image multimodal fusion and segmentation network is trained and validated using the training set and the validation set.

[0016] When the verification accuracy reaches the preset threshold, training stops, and the trained remote sensing image multimodal fusion segmentation network is obtained.

[0017] In addition to the aspects and any possible implementations described above, a further implementation is provided, which includes, before dividing the target remote sensing image set, category label set, and edge label set into a training set and a validation set according to a preset ratio:

[0018] The target remote sensing image set, category label set, and edge label set are cut to the same size.

[0019] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the pre-built remote sensing image multimodal fusion segmentation network consists of a U-shaped symmetric encoder-decoder structure;

[0020] The encoder part uses a hybrid Mamba-CNN model;

[0021] The decoder section includes: an edge-guided fusion module, a category-guided fusion module, and a salient feature filtering module.

[0022] In addition to the aspects and any possible implementations described above, another implementation is provided.

[0023] The edge-guided fusion module is constructed using the following formula:

[0024]

[0025] GateDown3 = Conv(Fus i )

[0026] Down i =Downsample(Fusi-1 )+Fus i GateDown i-1

[0027] GateDown i =Conv(Down i )

[0028] Up i =Upsample(Down i-1 )+Fus i

[0029] V i =GateUpi*Up i

[0030] Down i and Up i It is the intermediate state feature map output by the Feature Pyramid Network (FPN) and the Path Aggregation Network (PAN), GateDown. i and GateUp i It is the coarse edge segmentation map of the intermediate states output by FPN+PAN, V i These are the obtained coarse edge segmentation maps at different scales, Fus i This refers to the initial fusion of RGB and DSM features. Downsample and Upsample represent downsampling and upsampling operations, respectively, and Conv represents the convolution operation.

[0031] In addition to the aspects described above and any possible implementations, an implementation is further provided in which the category-guided fusion module is constructed by the following formula:

[0032] Q RGB K RGB =MLP(F RGB )

[0033] Q DSM K DSM =MLP(F DSM )

[0034]

[0035] Where F refers to the output of the previous layer decoder (if it is the first layer decoder, then F = F). RGB +F DSM ), F RGB F DSM This refers to the input RGB and DSM feature maps. MLP refers to a multilayer perceptron consisting of two convolutional layers. Q RGB K RGBThis refers to the feature maps obtained by processing RGB feature maps through an MLP, which are used as the Q and K matrices in self-attention, respectively. DSM K DSM This refers to the feature maps obtained by processing the DSM feature map through the MLP, which are used as the Q and K matrices in self-attention, respectively. Matrix refers to the self-attention matrix. and These refer to the output category enhancement and edge enhancement features, respectively. This is the final enhanced feature map output by the fusion module.

[0036] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the salient feature filtering module is constructed by the following formula:

[0037]

[0038] Where F refers to the input feature map. Y refers to the salient features output after multiple pooling layers. MaxPool and AvgPoo1 represent max pooling and average pooling, respectively. Y refers to the final output of the salient feature filtering module.

[0039] According to a second aspect of this disclosure, a multimodal fusion segmentation apparatus based on prior information and a Mamba mixture model is provided. The apparatus includes:

[0040] The data acquisition module is used to acquire RGB remote sensing image sets and DSM remote sensing image sets as the remote sensing image sets to be processed;

[0041] The data input module is used to input the set of remote sensing images to be processed into a pre-trained remote sensing image multimodal fusion and segmentation network;

[0042] The result output module is used to output the semantic segmentation results of the ground feature interpretation of the remote sensing image set to be processed.

[0043] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0044] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method as described in the first aspect of this disclosure.

[0045] The embodiments of this disclosure provide a multimodal fusion segmentation method and apparatus based on prior information and the Mamba hybrid model. While exploring the potential of the Mamba hybrid model in multimodal fusion, it combines prior information to effectively fuse RGB and DSM remote sensing images, thereby improving the accuracy of semantic segmentation of ground features in remote sensing images.

[0046] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0048] Figure 1 A flowchart of a multimodal fusion segmentation method based on prior information and a Mamba hybrid model according to an embodiment of the present disclosure is shown;

[0049] Figure 2 An overall block diagram of a remote sensing image multimodal fusion and segmentation network according to an embodiment of the present disclosure is shown;

[0050] Figure 3 A structural block diagram of an edge-guided fusion module according to an embodiment of the present disclosure is shown;

[0051] Figure 4 A structural block diagram of a category-guided fusion module according to an embodiment of the present disclosure is shown;

[0052] Figure 5 A structural block diagram of a salient feature filtering module according to an embodiment of the present disclosure is shown;

[0053] Figure 6 A block diagram of a multimodal fusion segmentation apparatus based on prior information and a Mamba hybrid model according to an embodiment of the present disclosure is shown;

[0054] Figure 7 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0056] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0057] Figure 1 A flowchart of a multimodal fusion segmentation method 100 based on prior information and a Mamba mixture model according to an embodiment of the present disclosure is shown. Method 100 includes:

[0058] Step 110: Obtain the RGB remote sensing image set and the DSM remote sensing image set as the remote sensing image set to be processed.

[0059] In some embodiments, a remote sensing image set Dataset0 to be processed is acquired, which includes: an RGB remote sensing image set RGB_Dataset0 and a DSM remote sensing image set DSM_Dataset0. The RGB remote sensing image fuses the wavelengths of the red, blue, and green spectral channels and records information from these three bands. The DSM remote sensing image originates from a DSM (Digital Surface Model), an elevation model that displays surface height. A DSM includes existing surface shapes, such as tree height, buildings, and any objects on the ground. DSMs can realistically represent the surface undulations and can be widely used in various industries.

[0060] Step 120: Input the set of remote sensing images to be processed into a pre-trained remote sensing image multimodal fusion and segmentation network.

[0061] In some embodiments, the remote sensing image multimodal fusion segmentation network is trained through the following steps: acquiring a target remote sensing image set; wherein the target remote sensing image set is a subset of the remote sensing image set to be processed; labeling the target remote sensing image set to obtain a category label set; extracting edges from the category label set to obtain an edge label set; dividing the target remote sensing image set, the category label set, and the edge label set into a training set and a validation set according to a preset ratio; training and validating the pre-constructed remote sensing image multimodal fusion segmentation network using the training set and the validation set; stopping training when the validation accuracy reaches a preset threshold, and obtaining the trained remote sensing image multimodal fusion segmentation network.

[0062] In some embodiments, it is necessary to pre-construct a multimodal fusion and segmentation network for remote sensing images, such as... Figure 2 The diagram shows the overall block diagram of the remote sensing image multimodal fusion and segmentation network. This network consists of a U-shaped symmetric encoder-decoder structure. The encoding part employs a Mamba-CNN hybrid model. The decoding part comprises three components: category-guided fusion, edge-guided fusion, and a salient feature filtering module, used to refine the segmentation map guided by the network. The decoder output is processed by two fully connected (MLP) layers to obtain the final output. In other words, based on the state-selection space Mamba method, global modeling can be achieved while maintaining a relatively small computational load, enabling global modeling of ultra-long sequences.

[0063] Among them, such as Figure 3 The diagram shows the structural block of the edge-guided fusion module. This module includes: coarse edge segmentation maps used in each stage are obtained by applying a "shape flow" gate structure to each stage of the encoder. By forcibly filtering semantic information, some branches of the multimodal fusion segmentation network encoder are forced to learn only edge information. Each stage of the encoder outputs a coarse edge segmentation map V1, V2, and V3, thus obtaining feature representations at different scales, specifically constructed using the following formula:

[0064]

[0065] GateDown3 = Conv(Fus i )

[0066] Down i =Downsample(Fus i-1 )+Fus i GateDown i-1

[0067] GateDown i =Conv(Down i )

[0068] Up i =Upsample(Down i-1 )+Fus i

[0069] V i =GateUp i *Up i

[0070] Down i and Up i It is an intermediate state feature map output by Feature Pyramid Network + Path Aggregation Network (FPN+PAN), GateDown i and GateUp i It is a coarse segmentation map of the intermediate state edges output by FPN+PAN, such as Figure 3 As shown; V i These are the obtained coarse edge segmentation maps at different scales, Fus i This refers to the initial fusion of RGB and DSM features. Downsample and Upsample represent downsampling and upsampling operations, respectively, and Conv represents the convolution operation.

[0071] Among them, such as Figure 4 The block diagram of the category-guided fusion module shown is as follows. The category-guided fusion module includes: considering the category information of the modality itself and across modalities, using a coarse category segmentation map to infer the importance of each modality, and making full use of prior category information to guide fusion. Specifically, it is constructed by the following formula:

[0072] Q RGB K RGB =MLP(F RGB )

[0073] Q DSM K DSM =MLP(F DSM )

[0074]

[0075] Where F refers to the output of the previous layer decoder (if it is the first layer decoder, then F = F). RGB +F DSM ), F RGB F DSM This refers to the input RGB and DSM feature maps. MLP refers to a multilayer perceptron consisting of two convolutional layers. Q RGB K RGB This refers to the feature maps obtained by processing RGB feature maps through an MLP, which are used as the Q and K matrices in self-attention, respectively. DSM K DSMThis refers to the feature maps obtained by processing the DSM feature map through the MLP, which are used as the Q and K matrices in self-attention, respectively. Matrix refers to the self-attention matrix. and These refer to the output category enhancement and edge enhancement features, respectively. This is the final enhanced feature map output by the fusion module.

[0076] Among them, such as Figure 5 The block diagram of the salient feature filtering module shown is provided. This module includes: removing some salient features through multi-level pooling, thereby enabling the multimodal fusion segmentation network to mine deep semantic information from the fusion of RGB and DSM modalities. This alleviates some of the erroneous classification shortcuts learned due to dataset bias, enhancing the network's actual segmentation ability. Specifically, it is constructed using the following formula:

[0077]

[0078] Where F refers to the input feature map. Y refers to the salient features output after multiple pooling layers. MaxPool and AvgPool represent max pooling and average pooling, respectively. Y refers to the final output of the salient feature filtering module.

[0079] In some embodiments, a subset of Dataset0, Dataset1 (RGB_Dataset1, DSM_Dataset1), is taken, and Dataset1 is labeled to obtain the class label set Dataset1_Class_Label. Further processing is then performed on this data set (using edge extraction methods for the class label set (including but not limited to using the Canny operator), or by defining K consecutive pixels in the class label set as different classes to obtain a rough square edge), resulting in the corresponding edge label set Dataset1_Edge_Label. The remote sensing image set Dataset1, the class label set, Dataset1_Class_Label, and the edge label set Dataset1_Edge_Label are then cut to a fixed size, i.e., the target remote sensing image set, class label set, and edge label set are cut to the same size. The dataset is divided into training and validation sets according to a certain ratio. After data augmentation, the training and validation sets are input into the remote sensing image multimodal fusion segmentation network constructed as described above for training. The parameter weights with the best accuracy in the validation phase are obtained, i.e., the training of the remote sensing image multimodal fusion segmentation network ends at this point. For example, the training set F (RGB and DSM remote sensing images) is input into the Mamba-CNN hybrid model encoder (RGB is input to the visual state space model (V-Mamba-Small), and DSM is input to the second version of the residual network (Res2Net-50)). Before each downsampling, the current feature representation is retained, resulting in five feature representations at different resolutions: E1, E2, E3, E4, and E5.

[0080]

[0081] Where F RGB This refers to the input 3-channel RGB image, F DSM This refers to the input 1-channel DSM image.

[0082] Secondly, the feature representation result E5 output from the deepest layer of the encoder is used as the input to the first layer D5 of the decoder, and then input to the salient feature filtering module to filter out some unimportant features.

[0083]

[0084] Here, Filter refers to the salient feature filtering module, and W1 and W2 are the learned adaptive weights that satisfy (W1+W2=1).

[0085] Then, in layers two through four of the decoder, the layer above the decoder D... i The feature representation of the output and encoder at the corresponding resolution E ii∈(2-5) is used as input and weighted bitwise addition is performed for fusion. The fusion result is divided into two branches, which use the category-guided fusion module and the edge-guided fusion module respectively. They start from high / low frequency (the category is regarded as a smooth low-frequency signal and the edge is regarded as a sharpened high-frequency signal) and fuse RGB and DSM remote sensing images.

[0086]

[0087] Here, Decoder refers to the decoder, D i -1 refers to the feature map output by the i-th layer of the decoder.

[0088] Finally, the output of the fourth layer of the decoder will be upsampled by a factor of two, fused with E1, then passed through two fully connected layers Mlp, and upsampled by a factor of two again to obtain the final segmentation result Output.

[0089]

[0090] Upsample refers to bilinear interpolation upsampling. W1, W2, and W3 are learned adaptive weights that satisfy (W1+W2+W3=1). The training and validation sets obtained from Datasetl are augmented and then input into the remote sensing image multimodal fusion segmentation network constructed as described above for training, thereby obtaining the parameter weights with the best accuracy in the validation stage.

[0091] In some embodiments, the remote sensing image set Dataset0 needs to be divided into the same size as the fixed size in the training process described above, and then input into the remote sensing image multimodal fusion and segmentation network.

[0092] Step 130: Output the semantic segmentation results of the ground feature interpretation for the remote sensing image set to be processed.

[0093] In some embodiments, by inputting a remote sensing image set Dataset0, which is divided into the same size as the one used in the training process described above, into the trained remote sensing image multimodal fusion segmentation network, a more accurate semantic segmentation result for the interpretation of ground features of the remote sensing image set Dataset0 can be obtained.

[0094] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0095] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0096] Figure 6 A block diagram of a multimodal fusion segmentation apparatus 600 based on prior information and a Mamba mixture model according to an embodiment of the present disclosure is shown. Figure 6 As shown, the device 600 includes:

[0097] Data acquisition module 610 is used to acquire RGB remote sensing image set and DSM remote sensing image set as remote sensing image set to be processed;

[0098] Data input module 620 is used to input the set of remote sensing images to be processed into a pre-trained remote sensing image multimodal fusion and segmentation network;

[0099] The result output module 630 is used to output the semantic segmentation results of the ground feature interpretation of the remote sensing image set to be processed.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0101] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0102] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0103] Electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in ROM 702 or a computer program loaded into RAM 703 from storage unit 708. RAM 703 can also store various programs and data required for the operation of electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. I / O interface 705 is also connected to bus 704.

[0104] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0105] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the multimodal fusion segmentation method based on prior information and the Mamba mixture model. For example, in some embodiments, the multimodal fusion segmentation method based on prior information and the Mamba mixture model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the multimodal fusion segmentation method based on prior information and the Mamba mixture model described above can be performed. Alternatively, in other embodiments, computing unit 701 may be configured by any other suitable means (e.g., by means of firmware) to perform a multimodal fusion segmentation method based on prior information and a Mamba hybrid model.

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0111] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0112] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0113] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multimodal fusion segmentation method based on prior information and a Mamba hybrid model, characterized in that, include: Acquire RGB and DSM remote sensing image sets as the remote sensing image sets to be processed; The set of remote sensing images to be processed is input into a pre-trained remote sensing image multimodal fusion and segmentation network; Output the semantic segmentation results of the ground cover interpretation for the remote sensing image set to be processed; The remote sensing image multimodal fusion segmentation network is trained through the following steps: acquiring a target remote sensing image set; wherein the target remote sensing image set is a subset of the remote sensing image set to be processed; labeling the target remote sensing image set to obtain a category label set; extracting edges from the category label set to obtain an edge label set; dividing the target remote sensing image set, the category label set, and the edge label set into a training set and a validation set according to a preset ratio; training and validating the pre-constructed remote sensing image multimodal fusion segmentation network using the training set and the validation set; stopping training when the validation accuracy reaches a preset threshold, thus obtaining the trained remote sensing image multimodal fusion segmentation network. The pre-built remote sensing image multimodal fusion and segmentation network consists of a U-shaped symmetrical encoder-decoder structure; wherein, the encoder part adopts a Mamba-CNN hybrid model; the decoder part includes: an edge-guided fusion module, a category-guided fusion module, and a salient feature filtering module; The edge-guided fusion module is constructed using the following formula: in and It is an intermediate state feature map output by the Feature Pyramid Network (FPN) and the Path Aggregation Network (PAN). and It is a coarse edge segmentation map of the intermediate states output by FPN+PAN. These are the obtained coarse edge segmentation maps at different scales. This refers to the initial fusion of RGB and DSM features. and These represent downsampling and upsampling operations, respectively, while Conv represents the convolution operation.

2. The method according to claim 1, characterized in that, Before dividing the target remote sensing image set, category label set, and edge label set into training set and validation set according to a preset ratio, the process includes: The target remote sensing image set, category label set, and edge label set are cut to the same size.

3. The method according to claim 1, characterized in that, The category-guided fusion module is constructed using the following formula: in This refers to the output of the previous layer decoder. If it is the first layer decoder, then , This refers to the input RGB and DSM feature maps. This refers to a multilayer perceptron consisting of two convolutional layers. This refers to the RGB feature map after... MLP The obtained feature maps are respectively used as self-attention... Q, K matrix ， This refers to the DSM feature map after... MLP The obtained feature maps are respectively used as self-attention... Q, K matrix, This refers to the self-attention matrix. and These refer to the output category enhancement and edge enhancement features, respectively. This is the final enhanced feature map output by the fusion module.

4. The method according to claim 1, characterized in that, The salient feature filtering module is constructed using the following formula: in This refers to the input feature map. This refers to the salient features output after multiple pooling processes. and These represent max pooling and average pooling, respectively. This refers to the final output of the salient feature filtering module.

5. A multimodal fusion segmentation device based on prior information and a Mamba hybrid model, characterized in that, The method for performing the multimodal fusion segmentation method based on prior information and the Mamba hybrid model as described in any one of claims 1-4 includes: The data acquisition module is used to acquire RGB remote sensing image sets and DSM remote sensing image sets as the remote sensing image sets to be processed; The data input module is used to input the set of remote sensing images to be processed into a pre-trained remote sensing image multimodal fusion and segmentation network; The result output module is used to output the semantic segmentation results of the ground feature interpretation of the remote sensing image set to be processed.

6. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the multimodal fusion segmentation method based on prior information and Mamba hybrid model as described in any one of claims 1-4.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the multimodal fusion segmentation method based on prior information and the Mamba hybrid model as described in any one of claims 1-4.

Images