Multimodal remote sensing image segmentation method, system and electronic device

Abstract

This disclosure presents a multimodal remote sensing image segmentation method, system, and electronic device. The method involves acquiring multimodal remote sensing images, preprocessing these images (including optical images and elevation images), extracting optical features from the optical images, and extracting elevation features from the elevation images. A graph Laplacian matrix is ​​constructed based on the optical and elevation features, and feature fusion is performed using this matrix to obtain target fusion features. A segmentation model is then invoked to predict the category probability based on the target fusion features, resulting in a predicted segmentation result for the multimodal remote sensing image. The target loss is determined based on the predicted segmentation result and the label segmentation result, and the segmentation model is trained based on this target loss. Finally, the trained segmentation model is invoked to segment the remote sensing image to be processed, resulting in a segmentation result for the remote sensing image to be processed. This method improves the segmentation accuracy and efficiency of multimodal remote sensing images.

Description

Technical Field

[0001] This disclosure relates to the field of image segmentation technology, and in particular to multimodal remote sensing image segmentation methods, systems and electronic devices. Background Technology

[0002] With the continuous development of remote sensing technology, multimodal remote sensing images, leveraging their complementary features from heterogeneous data, have become a crucial foundation for high-precision land cover classification. Current multimodal remote sensing image segmentation methods mostly employ convolutional neural networks or Transformers for feature alignment and fusion. Furthermore, to enhance the adaptability of the visual base model to multimodal remote sensing images, the LoRA algorithm is typically used for efficient local parameter fine-tuning. However, most current mainstream visual base models are only applicable to three-channel optical images, and parameter fine-tuning methods are generally limited to combinations of LoRa and adapters, exhibiting insufficient adaptability and limited generalization. This makes them unsuitable for the complex segmentation tasks of multimodal remote sensing images, thus hindering further improvements in segmentation accuracy. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail in this disclosure. This overview is not intended to limit the scope of the claims.

[0004] This disclosure provides a multimodal remote sensing image segmentation method that can improve the segmentation accuracy and efficiency of the segmentation model for multimodal remote sensing images.

[0005] On one hand, embodiments of this disclosure provide a multimodal remote sensing image segmentation method, including: Acquire multimodal remote sensing images and preprocess the multimodal remote sensing images, wherein the multimodal remote sensing images are acquired by various types of sensors, including optical images and elevation images; The optical image is subjected to feature extraction to obtain optical features, and the elevation image is subjected to feature extraction to obtain elevation features; A graph Laplacian matrix is ​​constructed based on the optical features and the elevation features. The graph Laplacian matrix is ​​adjusted according to a dynamic neighborhood weighted aggregation mechanism. Graph convolution is used to perform feature fusion based on the adjusted graph Laplacian matrix to obtain the target fused features. The segmentation model is invoked to predict the category probability based on the target fusion features. The predicted segmentation result of the multimodal remote sensing image is obtained based on the category probability prediction result. The target loss is determined based on the difference between the predicted segmentation result and the label segmentation result. The segmentation model is trained based on the target loss. During the training of the segmentation model, a double diagonal low-rank matrix is ​​added to the linear layer of the segmentation model. The double diagonal low-rank matrix is ​​updated based on the target loss. In response to an image segmentation request, the trained segmentation model is invoked to segment the remote sensing image to be processed, thereby obtaining the segmentation result of the remote sensing image to be processed.

[0006] On the other hand, embodiments of this disclosure also provide a multimodal remote sensing image segmentation system, including: The preprocessing module is used to acquire multimodal remote sensing images and preprocess the multimodal remote sensing images, wherein the multimodal remote sensing images are acquired by various types of sensors, including optical images and elevation images; The feature extraction module is used to extract features from the optical image to obtain optical features, and to extract features from the elevation image to obtain elevation features; The feature fusion module is used to construct a graph Laplacian matrix based on the optical features and the elevation features, adjust the graph Laplacian matrix according to a dynamic neighborhood weighted aggregation mechanism, and perform feature fusion using graph convolution based on the adjusted graph Laplacian matrix to obtain the target fused features. The model training module is used to call the segmentation model to perform category probability prediction based on the target fusion features, obtain the predicted segmentation result of the multimodal remote sensing image based on the category probability prediction result, determine the target loss based on the difference between the predicted segmentation result and the label segmentation result, and train the segmentation model based on the target loss. During the training of the segmentation model, a double diagonal low-rank matrix is ​​added to the linear layer of the segmentation model, and the double diagonal low-rank matrix is ​​updated based on the target loss. The double diagonal low-rank matrix includes two diagonal matrices and at least one low-rank matrix. The image segmentation module is used to respond to an image segmentation request by calling the trained segmentation model to segment the remote sensing image to be processed, and to obtain the segmentation result of the remote sensing image to be processed.

[0007] On the other hand, embodiments of this disclosure also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described multimodal remote sensing image segmentation method.

[0008] The embodiments disclosed herein include at least the following beneficial effects: By acquiring multimodal remote sensing images and preprocessing them, key information such as edges and textures in the multimodal remote sensing images can be effectively highlighted, laying a solid foundation for improving the accuracy of subsequent feature extraction. Next, feature extraction is performed on the optical image to obtain optical features, and feature extraction is performed on the elevation image to obtain elevation features. A graph Laplacian matrix is ​​constructed based on the optical and elevation features. Adjusting the graph Laplacian matrix according to a dynamic neighborhood weighted aggregation mechanism can improve the fusion efficiency of multimodal features. Then, graph convolution is used to fuse features based on the adjusted graph Laplacian matrix to obtain the target fused features. This allows for the establishment of long-distance dependencies between different modal features during graph convolution processing, further improving the fusion efficiency of different modal features. Finally, a segmentation model is called to predict the category probability based on the target fused features, and the predicted segmentation result of the multimodal remote sensing image is obtained based on the category probability prediction result. Based on the difference between the predicted segmentation result and the label segmentation result, the target loss is determined, and the segmentation model is trained based on the target loss. During the training process, a double diagonal low-rank matrix is ​​added to the linear layer of the segmentation model. The double diagonal low-rank matrix is ​​updated based on the target loss. The double diagonal low-rank matrix provides additional linear transformation capability to the segmentation model, improving the segmentation model's adaptability to different data distributions and thus enhancing its feature extraction capability. In response to an image segmentation request, the trained segmentation model is invoked to segment the remote sensing image to be processed, obtaining the segmentation result of the remote sensing image to be processed. Ultimately, this improves the segmentation accuracy and efficiency of the segmentation model for multimodal remote sensing images.

[0009] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing this disclosure. Attached Figure Description

[0010] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0011] Figure 1 An optional flowchart of the multimodal remote sensing image segmentation method provided in this disclosure embodiment; Figure 2 Provided for the embodiments of this disclosure Figure 1 An optional flowchart for step S101; Figure 3 This is an optional schematic diagram illustrating multimodal remote sensing image preprocessing provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of an optional structure of the segmentation model provided in an embodiment of the present disclosure; Figure 5 An optional schematic diagram of the first feature extraction module provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of an optional structure of the feature injector provided in an embodiment of the present disclosure; Figure 7 This is an optional schematic diagram of a feature fusion module provided in an embodiment of this disclosure; Figure 8 This is an optional schematic diagram illustrating the fusion process of the fusion submodule provided in an embodiment of this disclosure; Figure 9 Provided for the embodiments of this disclosure Figure 8 An optional flowchart of the fusion process of the fusion sub-modules in the middle; Figure 10 This is an optional schematic diagram of a double diagonal low-rank matrix provided in an embodiment of this disclosure; Figure 11 This is an optional training diagram of the second feature extraction module provided in an embodiment of this disclosure; Figure 12 A schematic diagram of an optional overall framework for the multimodal remote sensing image segmentation method provided in this disclosure embodiment. Figure 13 This is a schematic diagram of the structure of the multimodal remote sensing image segmentation system provided in the embodiments of this disclosure; Figure 14 This is a schematic diagram of an optional structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0013] It should be noted that in the various specific embodiments of this disclosure, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. The target object can be a user. In addition, when embodiments of this disclosure require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this disclosure be obtained.

[0014] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0015] To facilitate understanding of the technical solutions provided in the embodiments of this disclosure, some key terms used in the embodiments of this disclosure will be explained below: Multimodal remote sensing images refer to a set of complementary remote sensing data for the same observation area, acquired by different types of sensors, different imaging mechanisms, or different spectral bands. Common modalities include optical remote sensing images, elevation images, infrared images, and multispectral / hyperspectral images. Different modal data reflect the differentiated physical attributes of ground objects, such as spectral reflectance, spatial structure, elevation morphology, and scattering characteristics. Through the complementarity of heterogeneous information, multimodal data can provide more comprehensive and robust feature support for ground object recognition, scene analysis, and semantic segmentation than single-modal data.

[0016] LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning (PEFT) method. Its core idea is to introduce a pair of low-rank decomposition matrices alongside the weight matrix of the pre-trained model for training, while simultaneously freezing the main parameters of the model. By updating only a small number of low-rank parameters, the model can be adapted to downstream tasks, preserving the original feature representation capabilities of the pre-trained model while significantly reducing the number of training parameters and computational overhead.

[0017] Graph Convolutional Networks (GCNs) are a class of deep learning models for graph-structured data. Their core functionality involves aggregating and transferring feature information through adjacency relationships between nodes and the graph Laplacian matrix. GCNs extend the local weighting concept of traditional convolution to non-Euclidean graph structures, iteratively aggregating features from neighboring nodes to achieve global context modeling and long-distance dependency capture, effectively modeling the relationships between nodes.

[0018] With the continuous development of remote sensing technology, multimodal remote sensing images, leveraging the complementary features of heterogeneous data, have become an important foundation for high-precision land cover classification. In recent years, benefiting from the rapid development of computer vision technology, deep learning-based semantic segmentation algorithms have become an important tool for interpreting remote sensing images. However, due to the differences in the distribution of features across different modalities, multimodal remote sensing image semantic segmentation faces challenges such as difficulties in cross-modal feature fusion and alignment, and poor fusion results. In existing technologies, most multimodal remote sensing image segmentation methods employ convolutional neural networks or Transformers to achieve feature alignment and fusion. Furthermore, to improve the adaptability of the visual base model to multimodal remote sensing images, training typically relies on full-parameter fine-tuning, the use of multi-parameter efficient fine-tuning (PEFT) strategies, or the use of LoRA's locally efficient parameter fine-tuning strategy. However, most current mainstream vision models are only applicable to three-channel optical images. Furthermore, convolutional neural network-based fusion methods, due to their limited receptive field, cannot fully consider global contextual information when fusing multimodal features, making it difficult to establish long-term dependencies between modalities and resulting in insufficient modal feature interaction. Transformer-based fusion methods, on the other hand, have high computational complexity, limiting the model's inference speed. In addition, existing efficient parameter fine-tuning methods are generally limited to LoRa and adapter combinations, exhibiting insufficient adaptability and limited generalization, making them unsuitable for complex segmentation tasks of multimodal remote sensing images, thus hindering further improvements in segmentation accuracy.

[0019] Based on this, the present disclosure provides a multimodal remote sensing image segmentation method, which establishes the potential correlation between different modal features and obtains target fusion features that have multimodal, multi-scale integrity and feature representation accuracy.

[0020] Reference Figure 1 , Figure 1 This is an optional flowchart of a multimodal remote sensing image segmentation method provided in the embodiments of this disclosure. The multimodal remote sensing image segmentation method includes, but is not limited to, the following steps S101 to S105.

[0021] Step S101: Acquire multimodal remote sensing images and preprocess them.

[0022] Multimodal remote sensing images are acquired by various types of sensors, such as optical sensors, lidar, and synthetic aperture radar. These images include optical images, digital surface models (DSMs), and synthetic aperture radar (SAR) images. Optical sensors acquire optical images, which can specifically be RGB three-channel natural images. LiDAR acquires elevation information, which is then processed to generate elevation images, specifically digital surface models used to characterize the height of ground features. Synthetic aperture radar acquires synthetic aperture radar images, which characterize the scattering properties of ground features.

[0023] It is understood that the multimodal remote sensing images used as training samples in this disclosure consist of data from multiple modalities, and the pixel category labels in the multimodal remote sensing images have been labeled. For the multimodal remote sensing image segmentation method of this disclosure, the input multimodal remote sensing image can be represented as follows: k is the number of modes, and the output is the label Y. The input-output relationship can be defined by the following formula, where G(·) represents the segmentation network and m represents the category output by the segmentation network.

[0024]

[0025] Because the acquired multimodal remote sensing images are multi-channel images, they can be preprocessed and digitally augmented. (Refer to...) Figure 2 , Figure 2 Provided for the embodiments of this disclosure Figure 1 An optional flowchart for step S101 is provided, wherein the multimodal remote sensing image segmentation method may include, but is not limited to, steps S201 to S202.

[0026] Step S201: Acquire multimodal remote sensing images, and perform overlapping and segmentation processing on the multimodal remote sensing images based on a preset step size to obtain multiple remote sensing image blocks of fixed size.

[0027] Specifically, the preset step size is 256 pixels. An image overlap strategy is adopted, with a step size of 256 pixels. The multimodal remote sensing image is overlapped and cut through a sliding window to obtain multiple remote sensing pixel blocks of fixed size. The size of the remote sensing pixel block can be 512×512.

[0028] Step S202: Perform image enhancement on multiple remote sensing image patches, and stitch the enhanced remote sensing image patches together to obtain an enhanced multimodal remote sensing image with the same size as the preprocessed multimodal remote sensing image.

[0029] Specifically, geometric enhancement and pixel enhancement strategies are used to enhance multiple remote sensing image patches, resulting in enhanced remote sensing image patches of a fixed size. (Refer to...) Figure 3 , Figure 3 This is an optional schematic diagram of multimodal remote sensing image preprocessing provided in this embodiment. Geometric enhancement strategies include operations such as translation, scaling, cropping, rotation, horizontal flipping, and vertical flipping. Pixel enhancement strategies include operations such as brightness adjustment, saturation adjustment, and contrast adjustment. Next, multiple enhanced remote sensing image blocks are stitched together to obtain an enhanced multimodal remote sensing image with the same size as the original multimodal remote sensing image. It should be noted that the multimodal remote sensing images (including optical images and elevation images) involved in steps S102-S104 are all images after the aforementioned image enhancement processing.

[0030] In some embodiments, at least one of the above-described image enhancement operations is randomly applied to enhance the remote sensing image patch to obtain an enhanced remote sensing image patch of a fixed size. For example, the remote sensing image patch is rotated, horizontally / vertically flipped, its brightness adjusted, its saturation adjusted, and its contrast adjusted to obtain an enhanced remote sensing image patch of a fixed size of 512×512.

[0031] Step S102: Extract features from the optical image to obtain optical features, and extract features from the elevation image to obtain elevation features.

[0032] The multimodal remote sensing image segmentation method disclosed herein is implemented by a segmentation model, which is a basic model including a first feature extraction module and a second feature extraction module. The first feature extraction module is used to extract elevation images, and specifically, the first feature extraction module can be a depthwise separable convolution block (DSC). The second feature extraction module is used to extract features from optical images, and specifically, the second feature extraction module can be a multi-token prediction block (MTP) in a Transformer network.

[0033] In some embodiments, the second feature extraction module includes three Transformer blocks. The first two Transformer blocks employ a sliding window multi-head self-attention mechanism with a sliding window size of 7, and the third Transformer block employs a global multi-head self-attention mechanism. All three Transformer blocks have 12 attention heads. The sliding window self-attention mechanism enables rapid interaction between pixels in different windows, while the global multi-head self-attention mechanism enables global information interaction of optical features. Based on this, using the second feature extraction module to extract features from optical images can effectively capture local details and global spatial dependencies in the optical images, improving the representational capability of optical features and providing strong optical feature support for the injection and fusion of elevation features.

[0034] In one possible implementation, multiple first feature extraction modules sequentially extract features from the elevation image to obtain multi-level elevation features. Similarly, multiple second feature extraction modules sequentially extract features from the optical image to obtain multi-level optical features. The elevation features are then injected into the optical features of the same level to obtain injected features, which are used as input to the next level of second feature extraction modules.

[0035] Specifically, refer to Figure 4 , Figure 4 This is a schematic diagram of an optional structure of the segmentation model provided in an embodiment of this disclosure. The segmentation module includes seven first feature extraction modules (DSC), four second feature extraction modules (MTP), and four feature injectors (MFI). The optical image is... H represents the height of the optical image, and W represents the width of the optical image. The optical image is embedded through an embedding layer. Flatten and perform feature mapping to obtain embedded features Embedded features Feature extraction is performed sequentially through MTP_1, MTP_2, MTP_3, and MTP_4 to obtain optical features at different levels output by each module. j=1,2,3,4, where j represents the feature extraction module at the j-th level, for example... This is the optical feature output from the second-level MTP_2. The feature size and number of channels of the elevation features remain unchanged across different levels.

[0036] Elevation imagery The features are extracted sequentially through DSC_1, DSC_2, DSC_3, and DSC_4, resulting in optical features at different levels output by each module. , For example, the elevation feature output after DSC_1 can be represented as the number of channels. When performing feature extraction in the first feature extraction module, refer to... Figure 5 , Figure 5 This is an optional schematic diagram of a first feature extraction module provided in an embodiment of this disclosure, wherein the input features are fed into the first feature extraction module. First, it passes through a convolution kernel of size 3×3 and an output dimension of 2*. The depthwise separable convolution DSConv1 is used to process the input features. After high-dimensional representation, batch normalization (BN) and non-linear activation of the ReLU function are performed to obtain the first deep feature. First depth feature After performing a 3×3 depthwise separable convolution DSConv2 with a kernel size of 3×3, batch normalization and non-linear activation with the ReLU function are applied to obtain the second depth feature. The second depth features are connected via residual connections. Compared with the first depth feature Feature fusion is performed to obtain the output of the first feature extraction module. .

[0037] In one possible implementation, during the process of injecting elevation features of the same level into optical features to obtain injected features, the elevation features can be transformed to match the feature dimensions of the elevation features with those of the optical features. Channel attention processing is then performed on both the optical features and the transformed elevation features to obtain optical attention features and elevation attention features. Finally, the optical attention features and elevation attention features are fused together to obtain the injected features.

[0038] Specifically, such as Figure 4 As shown, elevation features are injected through the feature injector MFI_1. Inject optical features at the same level In the process, the injection features are obtained. Injecting features As input to MTP_2. (See reference...) Figure 6 , Figure 6 This is a schematic diagram of an optional structure of the feature injector provided in the embodiments of this disclosure, which injects elevation features of the same level. and optical characteristics Input feature injector, elevation feature First, the feature transformation is performed through a convolutional layer (not shown in the attached diagram). Within this layer, convolution, batch normalization, and ReLU non-linear activation are performed sequentially to transform the elevation features. In terms of feature dimensions, compared with optical features Matching, the processed elevation features are To achieve multimodal feature enhancement, channel attention processing is applied to elevation and optical features. The elevation and optical features are sequentially passed through a global pooling layer, a convolutional layer (Conv1), and a convolutional layer (Conv2), and then connected to the elevation features via residual connections. Optical characteristics Multiplying these features yields elevation attention features and optical attention features. Adding these two features together yields the injection features. The above process can be represented by the following formula: For channel attention formula,

[0039]

[0040] In the formula, X represents the elevation characteristic. or optical features , This represents the global average pooling operator. Represents the ReLU activation function. This represents the Sigmoid activation function. For elevation attention characteristics, For optical attention characteristics, This indicates a channel-by-channel multiplication operation. This indicates the composition of functions.

[0041] Step S103: Construct a graph Laplacian matrix based on optical and elevation features, adjust the graph Laplacian matrix according to the dynamic neighborhood weighted aggregation mechanism, and perform feature fusion based on the adjusted graph Laplacian matrix to obtain the target fused features.

[0042] Specifically, the segmentation model includes a feature fusion module ( Figure 4 The GCFF (Generic Character Set) module includes multiple fusion sub-modules, which are used for feature enhancement and feature fusion.

[0043] In one possible implementation, the process of obtaining the target fusion features can be referred to Figure 7 , Figure 7 This is an optional schematic diagram of a feature fusion module provided in an embodiment of this disclosure. The feature fusion module includes 12 fusion sub-modules. Optical features... The input is processed in the first fusion submodule FGCN1 to perform feature enhancement, resulting in optically enhanced features. Elevation features The input is processed in the second fusion submodule FGCN2 for feature enhancement to obtain elevation enhancement features. Optical enhancement features With elevation enhancement features Input the third fusion submodule FGCN3, construct a graph Laplacian matrix based on optical enhancement features and elevation enhancement features, adjust the graph Laplacian matrix according to the dynamic neighborhood weighted aggregation mechanism, and perform feature fusion based on the adjusted graph Laplacian matrix to obtain the first fused feature. The first fusion feature is dominated by optical features. Optical enhancement features are then incorporated. and elevation enhancement features The fourth fusion submodule FGCN4 is input, and a graph Laplacian matrix is ​​constructed based on optical enhancement features and elevation enhancement features. The graph Laplacian matrix is ​​adjusted according to a dynamic neighborhood weighted aggregation mechanism, and feature fusion is performed based on the adjusted graph Laplacian matrix to obtain the second fused feature. The second fusion feature is dominated by elevation features. Finally, the first fusion feature... The first enhanced fused feature is obtained by performing feature enhancement in the fifth fusion submodule FGCN5. ; the second fusion feature Feature enhancement is performed on the input six-fusion submodule FGCN6 to obtain the second enhanced fusion feature. The first enhanced fusion feature Second enhanced fusion features Add them together to obtain the target fusion features. .

[0044] Taking the third fusion submodule as an example, refer to Figure 8 , Figure 8 This is an optional schematic diagram of the fusion process of the fusion submodule provided in an embodiment of this disclosure. Based on this, refer to... Figure 9 , Figure 9 Provided for the embodiments of this disclosure Figure 8 An optional flowchart of the fusion process of the fusion submodule in the multimodal remote sensing image segmentation method may include, but is not limited to, steps S901 to S904.

[0045] Step S901: Perform channel compression on the optical enhancement features to obtain compressed optical features. Perform global feature extraction on the elevation enhancement features and then perform channel compression to obtain compressed elevation features.

[0046] Specifically, since optical enhancement features and elevation enhancement features have different feature dimensions, feature transformation is needed to match their feature dimensions. For example... Figure 8As shown, compressed optical features are obtained by channel compression of the optical enhancement features using a 1×1 convolution kernel and a Sigmoid activation function. After global feature extraction of the elevation enhancement features using global average pooling, channel compression of the global elevation features is then performed using a 1×1 convolution kernel and a Sigmoid activation function, resulting in compressed elevation features with the same feature dimensions as the compressed optical features. For example, the optical enhancement features are... The elevation enhancement feature is The compressed optical features obtained after processing can be The compressed elevation feature obtained after processing can be .

[0047] Step S902: Perform matrix multiplication based on compressed optical features and compressed elevation features to obtain the Graph Laplace matrix.

[0048] Specifically, matrix multiplication is performed based on compressed optical features and compressed elevation features. The graph Laplacian matrix is ​​obtained by aligning the features. The graph Laplacian matrix can be calculated using the following formula.

[0049] Corresponding to Figure 8 , To compress optical features, To compress elevation features, To handle the linear transformation matrix of compressed optical features, To handle the linear transformation matrix of compressed elevation features, This represents the global average pooling operator; Diag(·) represents the output diagonal matrix whose diagonal elements are the input vectors.

[0050] Step S903: Adjust the graph Laplacian matrix according to the dynamic neighborhood weighted aggregation mechanism, and normalize the adjusted graph Laplacian matrix to obtain the standard graph Laplacian matrix.

[0051] Specifically, the standard graphical Laplacian matrix can be calculated using the following formula.

[0052] Where D is a diagonal matrix, and the diagonal elements of the diagonal matrix are... That is, the diagonal element is The sum of the i-th row or the sum of the i-th column. I is the identity matrix, used to enhance residual joins. For a standardized matrix, This represents the operation of inverting a matrix and taking the square root of each element.

[0053] Step S904: Perform feature transformation on the optical enhancement features to obtain optical graph node features, and perform graph convolution operation based on the standard graph Laplacian matrix and optical graph node features to obtain the first fused feature.

[0054] Specifically, feature transformation is performed on the optical enhancement features to obtain the optical map node features. ,like Figure 8 As shown, graph convolution operations are performed based on the standard graph Laplacian matrix and optical graph node features. The graph convolution operation can be expressed by the following formula.

[0055] in, This represents the linear transformation matrix of the optical graph node features. By performing the above graph convolution operation on the standard graph Laplacian matrix and the optical graph node features, not only can long-range dependencies between different modal features be effectively established, but also feature information from optical and elevation modes can be effectively fused. Compared to traditional graph convolution methods, Figure 8 The graph convolution operation shown can dynamically adjust the graph Laplacian matrix of mode A by using the inner product of the features of nodes in mode B and the features of nodes in mode A while maintaining low computational complexity with a small number of channels.

[0056] Understandably, this disclosure Figure 8 Input of the fusion submodule , This does not specifically refer to optical enhancement features and elevation enhancement features; when the fusion submodule is used for feature fusion, , Features of two different modalities, as described above in the third fusion submodule, To compress optical features, To compress elevation features; in the fourth fusion submodule, To compress elevation features, To compress optical features. When the fusion submodule is used for feature enhancement. , For features of the same modality, the current modality features can be enhanced. For example, in the first fusion submodule, , All are optical enhancement features; in the second fusion submodule, , All of these are elevation enhancement features.

[0057] Step S104: Call the segmentation model to perform category probability prediction based on target fusion features, obtain the predicted segmentation result of the multimodal remote sensing image based on the category probability prediction result, determine the target loss based on the difference between the predicted segmentation result and the label segmentation result, and train the segmentation model based on the target loss.

[0058] During the training of the segmentation model, the original weight matrix of the segmentation model is frozen, a double diagonal low-rank matrix is ​​added to the linear layer of the segmentation model, and the double diagonal low-rank matrix is ​​updated based on the target loss. The double diagonal low-rank matrix consists of two diagonal matrices and two low-rank matrices, and both the diagonal matrices and the low-rank matrices are trainable parameters.

[0059] In one possible implementation, during the process of determining the target loss based on the difference between the predicted segmentation result and the label segmentation result, and training the segmentation model based on the target loss, cross-entropy loss and overlap loss are determined based on the difference between the predicted segmentation result and the label segmentation result. The target loss is then determined based on the cross-entropy loss and the overlap loss, and the segmentation model is trained based on the target loss. Specifically, the cross-entropy loss is used to characterize pixel-level loss, and the overlap loss is a Dice loss used to characterize category-level loss.

[0060] Furthermore, since the segmentation model's processing layers include intermediate layers and output layers, in one possible implementation, during the process of determining cross-entropy loss and overlap loss based on the difference between the predicted segmentation result and the label segmentation result, intermediate features of the intermediate layer output are obtained, intermediate segmentation results are obtained based on these intermediate features, a first cross-entropy sub-loss and a first overlap sub-loss are determined based on the difference between the intermediate segmentation result and the label segmentation result, output features of the output layer are obtained, predicted segmentation results are obtained based on these output features, a second cross-entropy sub-loss and a second overlap sub-loss are determined based on the difference between the predicted segmentation result and the label segmentation result, intermediate layer loss terms are determined based on the first cross-entropy sub-loss and the first overlap sub-loss, output layer loss terms are determined based on the second cross-entropy sub-loss and the second overlap sub-loss, loss weights are assigned to the intermediate layer loss terms, the intermediate layer loss terms are weighted based on these loss weights, and the target loss is determined based on the weighted intermediate layer loss terms and the output layer loss terms. Here, the first cross-entropy sub-loss is the cross-entropy loss of the intermediate layer, the second cross-entropy sub-loss is the cross-entropy loss of the output layer, the first overlap sub-loss is the overlap loss of the intermediate layer, and the second overlap sub-loss is the overlap loss of the output layer.

[0061] Based on the above explanation, the target loss can be calculated using the following formula, where, For the first cross-entropy sub-loss, For the second cross-entropy sub-loss, For the first degree of overlap sub-loss, For the second overlap sub-loss, This is the loss weight.

[0062]

[0063] By combining the features of the intermediate layer and the output layer to calculate the target loss, multi-scale and multi-granularity supervision signals can be introduced. This not only effectively improves the stability of gradient backpropagation and the training convergence speed, but also guides the intermediate layer to learn more discriminative feature representations, achieving deep fusion of shallow detailed features and deep semantic features. This enhances the generalization ability and robustness of the segmentation model to different data distributions, thereby improving the segmentation accuracy of multimodal remote sensing images.

[0064] In one possible implementation, the construction of the doubly diagonal low-rank matrix involves specifically obtaining the original weight matrix of the linear layer of the segmentation model, decomposing it to obtain a first low-rank matrix and a second low-rank matrix, and determining the low-rank adapted matrix based on the original weight matrix, the first low-rank matrix, and the second low-rank matrix. Next, the first diagonal matrix and the second diagonal matrix are obtained, and matrix multiplication is performed on them in the order of first diagonal matrix, low-rank adapted matrix, and second diagonal matrix, i.e., left-multiplying and right-multiplying the low-rank adapted matrix respectively, to obtain the doubly diagonal low-rank matrix. (Refer to...) Figure 10 , Figure 10 This is an optional schematic diagram of a double-diagonal low-rank matrix provided in an embodiment of this disclosure, combined with... Figure 10 A doubly diagonal low-rank matrix can be represented by the following formula.

[0065] in, This is the first diagonal matrix. This is the second diagonal matrix. It is a low-rank post-fit matrix. This is the original weight matrix. To train stable parameters, their value can be set to 1. First low-rank matrix. It follows a mean of 0 and a variance of . The second low-rank matrix The value of is 0. After matrix multiplication of the first low-rank matrix A and the second low-rank matrix B, a matrix of rank r is obtained. By introducing two additional diagonal matrices, the dynamic change capability of the low-rank adapted matrix can be effectively improved, thereby enhancing the adaptability of the segmentation model to new data distributions and improving the feature extraction capability of the segmentation model.

[0066] During the training of the segmentation model, such as Figure 4 As shown, the black hexagonal symbol is a freeze marker, freezing the original weight matrices of all second feature extraction modules MTP1-4, and adding a double diagonal low-rank matrix to all linear layers of the second feature extraction module of the segmentation model. (Refer to...) Figure 11 , Figure 11This is an optional training diagram of the second feature extraction module provided in this embodiment, illustrating the forward propagation process of the second feature extraction module during training. The second feature extraction module includes a multi-head attention mechanism and a multilayer perceptron (MLP). The multi-head attention mechanism includes a query linear layer, a key linear layer, and a value linear layer, while the MLP includes a first perceptron linear layer and a second perceptron linear layer. Taking a single self-attention head in the multi-head attention mechanism of the second feature extraction module as an example, the input features are processed by self-attention in the self-attention module to obtain the attention head features output by the self-attention head. The attention head features are then added to the input features through a residual connection to obtain the attention features output by the multi-head attention mechanism. The calculation of these features can be expressed by the following formula.

[0067]

[0068]

[0069] Where Q is the query matrix, K is the key matrix, and V is the value matrix. , , To query the linear transformation weight matrices corresponding to the key-linear layer, key-linear layer, and value-linear layer, the input feature X is transformed into a query matrix, key matrix, and value matrix using these linear transformation weight matrices. During training, this is achieved based on a double-diagonal low-rank matrix. , , Conduct training. Standardized operations at the presentation layer. For attention head features, This represents the matrix transpose operation, where m is the feature dimension of the input feature X, and Z is the attention feature. It should be noted that in a multi-head attention mechanism, the output attention feature is obtained by fusing the attention head features from the outputs of multiple sub-attention heads; however, this disclosure omits a detailed description of this fusion process.

[0070] Next, the processing of attention feature Z by the multilayer perceptron can be represented by the following formula, where, This represents the GeLu activation function. This is the weight matrix corresponding to the first linear perception layer. The weight matrix corresponding to the second perceptual linear layer is used during training based on a double diagonal low-rank matrix. , Training is performed. Y is the output of the second feature extraction module.

[0071]

[0072]

[0073] Understandable Figure 10 and Figure 11 The parameters in the gray area are trainable parameters, while the parameters in the other white areas are frozen parameters. Trainable parameters are used in training, while frozen parameters are not used in training.

[0074] Step S105: In response to the image segmentation request, the trained segmentation model is invoked to segment the remote sensing image to be processed, and the segmentation result of the remote sensing image to be processed is obtained.

[0075] Specifically, the remote sensing image to be processed is a multimodal remote sensing image. A trained segmentation model is used to segment the image, and a feature fusion module fuses the multimodal features to obtain fused features. (Refer to...) Figure 4 , for injected features After performing a 1 / 16 upsampling operation, the feature is concatenated with the fused feature. The resulting concatenated feature is then input into DSC_5 to obtain the first output feature; the injected feature... After performing a 1 / 8 upsampling operation, the feature is concatenated with the first output feature. The resulting concatenated feature is then input into DSC_6 to obtain the second output feature; the injected feature... After performing a 1 / 4 upsampling operation, the feature is concatenated with the second output feature. The concatenated feature is then input into DSC_7 to obtain the third output feature. The probability distribution of each category corresponding to the third output feature is calculated, and the segmentation result of the remote sensing image to be processed is obtained based on the probability distribution of the categories.

[0076] Reference Figure 12 , Figure 12 This is a schematic diagram of an optional overall framework for the multimodal remote sensing image segmentation method provided in this disclosure. The multimodal remote sensing image segmentation method of this disclosure can be applied to ground feature segmentation scenarios. The following is a complete description of the principle of the multimodal remote sensing image segmentation method in this disclosure: The multimodal remote sensing image segmentation method provided in this disclosure is implemented by a segmentation model, which includes multiple first feature extraction modules, multiple second feature extraction modules, multiple feature injectors, and a feature fusion module. During training of the segmentation model, the original weight matrices of all second feature extraction modules are frozen. A double-diagonal low-rank matrix is ​​added to the second feature extraction modules. The double-diagonal low-rank matrix consists of two diagonal matrices and two low-rank matrices. Based on the original weight matrix of the segmentation model, the first low-rank matrix, and the second low-rank matrix, a low-rank adapted matrix is ​​formed. The first diagonal matrix and the second diagonal matrix are then left-multiplied and right-multiplied respectively with the low-rank adapted matrix to obtain the double-diagonal low-rank matrix.

[0077] First, the segmentation model is trained. A segmentation dataset is obtained for training, which contains multiple multimodal remote sensing images. The remote sensing images undergo preprocessing such as overlap segmentation and image enhancement to obtain enhanced multimodal remote sensing images, including optical images and elevation images.

[0078] Next, the elevation image is used to extract features through the first feature extraction module, resulting in multi-level elevation features. This first feature extraction module is specifically a depthwise separable convolutional block. Optical features are then extracted through the second feature extraction module, resulting in multi-level optical features. This second feature extraction module is specifically the MTP module within the trained Transformer base model. During feature extraction, elevation features of the same level are injected into the optical features using a feature injector, resulting in injected features. These injected features are then input into the next level of the second feature extraction module for further feature extraction. It is important to note that the second feature extraction module uses a parameter-updated double diagonal low-rank matrix to process the optical features.

[0079] After feature extraction at all levels is completed, the elevation and optical features output from the last first and second feature extraction modules are input into the feature fusion module. A graph Laplacian matrix is ​​constructed based on the elevation and optical features. This matrix is ​​used to model the correlation between features from different modalities. Then, graph convolution operations are used to align and fuse multimodal features, resulting in the target fused feature. Based on the target fused feature and the injected features from each level, the output features of the segmentation model's output layer are obtained. The predicted segmentation result is then derived from these output features. The target loss is determined based on the difference between the predicted and labeled segmentation results. This target loss is composed of cross-entropy loss and overlap loss. The segmentation model is then trained based on this target loss.

[0080] Finally, in response to the image segmentation request, the trained segmentation model is invoked to segment the remote sensing image to be processed, and the segmentation result of the remote sensing image to be processed is obtained.

[0081] It should also be noted that the multimodal remote sensing image segmentation method provided in this disclosure is essentially segmentation based on the fusion of different modal features. Therefore, this disclosure uses optical images (optical modality) and elevation images (elevation / topography modality) to illustrate the multimodal remote sensing image segmentation method. In practical applications, multimodal remote sensing image segmentation can also be performed based on optical images and synthetic aperture radar images. This disclosure does not specifically limit the modality type.

[0082] In summary, the multimodal remote sensing image segmentation method provided in this disclosure, by constructing a double-diagonal low-rank matrix and introducing two additional diagonal matrices on top of the existing low-rank adapted matrix, effectively enhances the dynamic change capability of the low-rank adapted matrix, strengthens the adaptability of the segmentation model to new data distributions, and thus improves the feature extraction capability of the segmentation model. Furthermore, in the feature fusion module, the compressed features of one modality are used to align the graph Laplacian matrix of another modality, and the graph Laplacian matrix is ​​dynamically adjusted through modality information alignment. This method not only improves the computation speed of the graph Laplacian matrix but also increases the fusion efficiency of different modal features. Simultaneously, it effectively establishes long-distance dependencies between modalities during graph convolution, achieving rapid modality fusion and effective modality interaction, ultimately improving the segmentation accuracy and efficiency of multimodal remote sensing images.

[0083] Reference Figure 13 , Figure 13 This is a schematic diagram of the structure of the multimodal remote sensing image segmentation system provided in this embodiment of the disclosure. The fusion feature construction system 1300 includes: The preprocessing module 1301 is used to acquire multimodal remote sensing images and preprocess them. The multimodal remote sensing images are acquired by various types of sensors, including optical images and elevation images. The feature extraction module 1302 is used to extract features from optical images to obtain optical features and to extract features from elevation images to obtain elevation features. The feature fusion module 1303 is used to construct a graph Laplacian matrix based on optical features and elevation features, adjust the graph Laplacian matrix according to a dynamic neighborhood weighted aggregation mechanism, and perform feature fusion using graph convolution based on the adjusted graph Laplacian matrix to obtain the target fused features. The model training module 1304 is used to call the segmentation model to perform category probability prediction based on target fusion features, obtain the predicted segmentation result of the multimodal remote sensing image based on the category probability prediction result, determine the target loss based on the difference between the predicted segmentation result and the label segmentation result, and train the segmentation model based on the target loss. During the training of the segmentation model, a double diagonal low-rank matrix is ​​added to the linear layer of the segmentation model, and the double diagonal low-rank matrix is ​​updated based on the target loss. The double diagonal low-rank matrix includes two diagonal matrices and at least one low-rank matrix. The image segmentation module 1305 is used to respond to an image segmentation request by calling the trained segmentation model to segment the remote sensing image to be processed, and to obtain the segmentation result of the remote sensing image to be processed.

[0084] Reference Figure 14 , Figure 14An optional structural diagram of an electronic device provided in this disclosure includes: at least one processor 1410; at least one memory 1420 for storing at least one program; in this embodiment, when at least one program is executed by at least one processor 1410, the multimodal remote sensing image segmentation method of the preceding embodiment can be executed.

[0085] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate to describe embodiments of this disclosure, for example, those that can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0086] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0087] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0088] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0091] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0093] The above is a detailed description of the preferred embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A multimodal remote sensing image segmentation method, characterized in that, include: Acquire multimodal remote sensing images and preprocess the multimodal remote sensing images, wherein the multimodal remote sensing images are acquired by various types of sensors, including optical images and elevation images; The optical image is subjected to feature extraction to obtain optical features, and the elevation image is subjected to feature extraction to obtain elevation features; A graph Laplacian matrix is ​​constructed based on the optical features and the elevation features. The graph Laplacian matrix is ​​adjusted according to a dynamic neighborhood weighted aggregation mechanism. Graph convolution is used to perform feature fusion based on the adjusted graph Laplacian matrix to obtain the target fused features. The segmentation model is invoked to predict the category probability based on the target fusion features. The predicted segmentation result of the multimodal remote sensing image is obtained based on the category probability prediction result. The target loss is determined based on the difference between the predicted segmentation result and the label segmentation result. The segmentation model is trained based on the target loss. During the training of the segmentation model, a double diagonal low-rank matrix is ​​added to the linear layer of the segmentation model. The double diagonal low-rank matrix is ​​updated based on the target loss. In response to an image segmentation request, the trained segmentation model is invoked to segment the remote sensing image to be processed, thereby obtaining the segmentation result of the remote sensing image to be processed.

2. The multimodal remote sensing image segmentation method according to claim 1, characterized in that, The segmentation model includes a feature fusion module, which comprises multiple fusion sub-modules for feature enhancement and feature fusion. The model involves constructing a graph Laplacian matrix based on the optical and elevation features, adjusting the graph Laplacian matrix according to a dynamic neighborhood weighted aggregation mechanism, and performing feature fusion based on the adjusted graph Laplacian matrix to obtain the target fused features, including: The optical features and the elevation features are respectively input into the first fusion submodule and the second fusion submodule for feature enhancement to obtain optical enhancement features and elevation enhancement features; The optical enhancement features and elevation enhancement features are input into the third fusion submodule. A graph Laplacian matrix is ​​constructed based on the optical features and the elevation features. The graph Laplacian matrix is ​​adjusted according to the dynamic neighborhood weighted aggregation mechanism. Feature fusion is performed based on the adjusted graph Laplacian matrix to obtain the first fused feature, which is dominated by the optical features. The optical enhancement features and elevation enhancement features are input into the fourth fusion submodule. A graph Laplacian matrix is ​​constructed based on the optical features and the elevation features. The graph Laplacian matrix is ​​adjusted according to the dynamic neighborhood weighted aggregation mechanism. Feature fusion is performed based on the adjusted graph Laplacian matrix to obtain the second fused feature, which is dominated by the elevation features. The first fusion feature and the second fusion feature are respectively input into the fifth fusion submodule and the sixth fusion submodule for feature enhancement to obtain the first enhanced fusion feature and the second enhanced fusion feature. The first enhanced fusion feature and the second enhanced fusion feature are added together to obtain the target fusion feature.

3. The multimodal remote sensing image segmentation method according to claim 2, characterized in that, The optical enhancement features and elevation enhancement features are input into the third fusion submodule. A graph Laplacian matrix is ​​constructed based on the optical features and the elevation features. The graph Laplacian matrix is ​​adjusted according to a dynamic neighborhood weighted aggregation mechanism. Feature fusion is performed based on the adjusted graph Laplacian matrix to obtain the first fused feature, including: The optical enhancement features are subjected to channel compression to obtain compressed optical features. The elevation enhancement features are subjected to global feature extraction and then channel compression to obtain compressed elevation features. Matrix multiplication is performed based on the compressed optical features and the compressed elevation features to obtain the Thulaplac matrix. The graph Laplacian matrix is ​​adjusted according to the dynamic neighborhood weighted aggregation mechanism, and the adjusted graph Laplacian matrix is ​​normalized to obtain the standard graph Laplacian matrix. The optical features are transformed to obtain optical graph node features. Based on the standard graph Laplacian matrix and the optical graph node features, a graph convolution operation is performed to obtain the first fused feature.

4. The multimodal remote sensing image segmentation method according to claim 1, characterized in that, The double-diagonal low-rank matrix comprises two diagonal matrices and two low-rank matrices, and the multimodal remote sensing image segmentation method further includes: Obtain the original weight matrix of the linear layer of the segmentation model, decompose the original weight matrix to obtain a first low-rank matrix and a second low-rank matrix, and determine the low-rank adapted matrix based on the original weight matrix, the first low-rank matrix and the second low-rank matrix. Obtain the first diagonal matrix and the second diagonal matrix, and perform matrix multiplication on the first diagonal matrix, the low-rank adapted matrix, and the second diagonal matrix in the order of the first diagonal matrix, the low-rank adapted matrix, and the second diagonal matrix to obtain a double diagonal low-rank matrix.

5. The multimodal remote sensing image segmentation method according to claim 1, characterized in that, The step of determining the target loss based on the difference between the predicted segmentation result and the label segmentation result, and training the segmentation model based on the target loss, includes: Based on the difference between the predicted segmentation result and the label segmentation result, the cross-entropy loss and the overlap loss are determined, and the target loss is determined based on the cross-entropy loss and the overlap loss.

6. The multimodal remote sensing image segmentation method according to claim 5, characterized in that, The segmentation model's processing layer includes an intermediate layer and an output layer. The step of determining cross-entropy loss and overlap loss based on the difference between the predicted segmentation result and the label segmentation result, and determining the target loss based on the cross-entropy loss and the overlap loss, includes: Obtain intermediate features from the intermediate layer output, obtain intermediate segmentation results based on the intermediate features, and determine the first cross-entropy sub-loss and the first overlap sub-loss based on the difference between the intermediate segmentation results and the label segmentation results; Obtain the output features of the output layer, obtain the predicted segmentation result based on the output features, and determine the second cross-entropy sub-loss and the second overlap sub-loss based on the difference between the predicted segmentation result and the label segmentation result. An intermediate layer loss term is determined based on the first cross-entropy sub-loss and the first overlap sub-loss. An output layer loss term is determined based on the second cross-entropy sub-loss and the second overlap sub-loss. Loss weights are configured for the intermediate layer loss terms. The intermediate layer loss terms are weighted based on the loss weights. The target loss is determined based on the weighted intermediate layer loss terms and the output layer loss terms.

7. The multimodal remote sensing image segmentation method according to claim 1, characterized in that, The segmentation model includes multiple first feature extraction modules and multiple second feature extraction modules. The first feature extraction modules are used to extract features from the elevation image, and the second feature extraction modules are used to extract features from the optical features. The process of extracting features from the optical image to obtain optical features and extracting features from the elevation image to obtain elevation features includes: Based on multiple first feature extraction modules, feature extraction is performed sequentially on the elevation image to obtain multi-level elevation features; based on multiple second feature extraction modules, feature extraction is performed sequentially on the optical image to obtain multi-level optical features. The elevation feature is injected into the optical feature at the same level to obtain the injected feature, and the injected feature is used as the input of the second feature extraction module at the next level.

8. The multimodal remote sensing image segmentation method according to claim 7, characterized in that, The step of injecting the elevation features of the same level into the optical features to obtain the injected features includes: The elevation feature is transformed to match the feature dimension of the elevation feature with the optical feature; Channel attention processing is performed on the optical features and the elevation features after feature transformation to obtain optical attention features and elevation attention features, respectively. The optical attention features and the elevation attention features are fused to obtain the injection features.

9. A multimodal remote sensing image segmentation system, characterized in that, include: The preprocessing module is used to acquire multimodal remote sensing images and preprocess the multimodal remote sensing images, wherein the multimodal remote sensing images are acquired by various types of sensors, including optical images and elevation images; The feature extraction module is used to extract features from the optical image to obtain optical features, and to extract features from the elevation image to obtain elevation features; The feature fusion module is used to construct a graph Laplacian matrix based on the optical features and the elevation features, adjust the graph Laplacian matrix according to a dynamic neighborhood weighted aggregation mechanism, and perform feature fusion using graph convolution based on the adjusted graph Laplacian matrix to obtain the target fused features. The model training module is used to call the segmentation model to perform category probability prediction based on the target fusion features, obtain the predicted segmentation result of the multimodal remote sensing image based on the category probability prediction result, determine the target loss based on the difference between the predicted segmentation result and the label segmentation result, and train the segmentation model based on the target loss. During the training of the segmentation model, a double diagonal low-rank matrix is ​​added to the linear layer of the segmentation model, and the double diagonal low-rank matrix is ​​updated based on the target loss. The double diagonal low-rank matrix includes two diagonal matrices and at least one low-rank matrix. The image segmentation module is used to respond to an image segmentation request by calling the trained segmentation model to segment the remote sensing image to be processed, and to obtain the segmentation result of the remote sensing image to be processed.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the multimodal remote sensing image segmentation method according to any one of claims 1 to 8.

Images