A multi-modal remote sensing image semantic segmentation method based on geometric prior guidance

Abstract

The application discloses a multi-modal remote sensing image semantic segmentation method based on geometric prior guidance, and relates to the technical field of remote sensing image processing. Visible light images and digital surface models are input into a segmentation network of an encoding-decoding structure to output semantic segmentation results. The network encoder comprises three branches, two symmetric ResNet main branches respectively extract bimodal multi-scale features, and a geometry branch constructs pixel-level three-dimensional coordinates from the digital surface model to extract geometric prior features. Each encoding stage relies on the geometric prior to quantify the spatial response difference of the bimodal, adaptively untangles and fuses various features. The deep network uses a state space model to complete cross-modal global modeling through a progressive strategy. The decoder restores the resolution by using a jump connection, and jointly optimizes the semantic segmentation and boundary detection tasks. The application can effectively suppress cross-modal interference, significantly improve the semantic segmentation accuracy of remote sensing images on the basis of maintaining linear computational complexity.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and computer vision technology, and in particular relates to a multimodal remote sensing image semantic segmentation method based on geometric prior guidance. Background Technology

[0002] High-precision semantic segmentation of remote sensing data is of great significance for Earth science research and related applications. In recent years, with the rapid development of high-resolution satellite and UAV imagery, the acquisition of multimodal remote sensing data (including optical imagery, multispectral and hyperspectral imagery, synthetic aperture radar (SAR) and lidar (LiDAR) data) has become more convenient. Effective integration of this multimodal data can provide more comprehensive surface information for Earth science tasks such as change detection, land cover mapping, and target extraction. Among these, semantic segmentation, as a pixel-by-pixel classification task, aims to classify each pixel into a specific land cover category, and has therefore received widespread attention.

[0003] Existing remote sensing semantic segmentation methods can be mainly divided into three categories: The first category is based on traditional machine learning methods, such as random forests, support vector machines, and conditional random fields. However, these methods are limited by their weak high-level semantic representation capabilities and are difficult to effectively extract discriminative features in complex scenes. The second category is based on convolutional neural networks (CNNs). Although significant progress has been made, convolution operations usually rely on limited local receptive fields for information aggregation. As a result, these methods are still insufficient in modeling long-distance dependencies and are better at capturing local details, but are difficult to fully model global contextual information. The third category is based on visual Transformers (ViTs). These methods model the relationship between any two positions in the feature map through a self-attention mechanism, which significantly enhances the ability to represent global contextual information. However, the secondary computational complexity of Transformers makes them face high computational costs in high-resolution remote sensing scenes, and their ability to represent small targets and fine-grained boundaries is still insufficient.

[0004] In terms of multimodal fusion, existing methods mainly face the following challenges: First, most existing methods neglect the interference problem between modes when fusing across modes, and use simple stitching or element-wise addition for early fusion, which makes it difficult to effectively handle statistical heterogeneity and noise differences between modes; Second, in high-resolution remote sensing image segmentation tasks, they often face the problem of blurred boundaries and insufficient multi-scale ground object perception capabilities; Third, most existing Mamba-based remote sensing segmentation methods are designed for single-modality applications, directly applying one-dimensional scanning strategies for text sequences to remote sensing images, which makes it difficult to effectively characterize the complex two-dimensional spatial relationships between pixels; Fourth, the three-dimensional geometric information contained in the digital surface model (DSM) is not fully utilized, and existing methods often simply use the DSM as an additional input channel, lacking an explicit geometric prior guidance mechanism.

[0005] Therefore, how to provide a remote sensing image segmentation method that can effectively integrate multimodal remote sensing data, explicitly introduce geometric priors, and achieve high-precision semantic segmentation while maintaining computational efficiency is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a multimodal remote sensing image semantic segmentation method guided by geometric priors. This method effectively integrates multimodal remote sensing data, explicitly introduces geometric priors, and achieves high-precision semantic segmentation while maintaining computational efficiency. It can explicitly introduce three-dimensional geometric priors to suppress cross-modal interference and improve remote sensing image segmentation accuracy while maintaining linear computational complexity.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a multimodal remote sensing image semantic segmentation method based on geometric prior guidance, comprising the following steps: S1. After normalizing the remote sensing image to be segmented and its corresponding digital surface model data, slice it into multiple image block pairs using a sliding window method. S2. Input each image patch into the constructed symmetric dual-branch convolutional neural network backbone encoder. Both the visible light backbone branch and the digital surface model backbone branch use ResNet as the backbone network to extract multi-scale features of the visible light mode and the digital surface model mode respectively. The two branches are symmetrical and each contains four levels, and output shallow features of each scale respectively. S3. Using the geometric position prior module, pixel-level three-dimensional coordinates are constructed from the digital surface model data and multi-scale three-dimensional geometric position features are extracted. The geometric position prior module constitutes a third coding branch independent of the symmetrical dual-branch backbone encoder. S4. In each encoding stage, guided by the geometric position features, the response differences between visible light features and digital surface model features to the three-dimensional spatial structure are quantified, and the shared features and modality-specific features are adaptively de-entangled and weighted fused to generate multi-scale fused features. S5. Flatten the deepest fusion feature in the multi-scale fusion features into a sequence and input it into the deep feature fusion module. Based on the state space model, the extraction and fusion of multi-modal deep global context features are completed through three progressive stages: selective disentanglement, cross-modal mutual enhancement, and bidirectional fusion. S6. Input the fused features output by the deep feature fusion module into the cascade decoder, restore the spatial resolution layer by layer through skip connections, jointly optimize the semantic segmentation and boundary detection tasks, and output the final semantic segmentation result.

[0008] Furthermore, in step S3, the operation of the geometric position prior module includes the following steps: S31. For the pixel located in the i-th row and j-th column of the digital surface model image, construct a three-dimensional coordinate system with coordinates (x, y, j). i,j ,y i,j ,h i,j ), where x i,j With y i,j These represent the normalized row and column positions of the pixel in the two-dimensional plane, respectively, h. i,j This is the normalized elevation value at this pixel, with all three components mapped to the [0,1] interval; S32. Stack the three-dimensional coordinates of all pixels in the image in spatial order to construct a three-dimensional coordinate tensor with a size of H×W×3, where H and W are the height and width of the input image, respectively. S33. The three-dimensional coordinate tensor is fed into a four-stage hierarchical convolutional encoder. Each stage uses a convolutional layer with a kernel size of 3×3, a stride of 2, and padding of 1 for downsampling, followed by a ReLU activation function to progressively extract multi-scale geometric position features; The spatial resolution of the geometric position features output by each stage is strictly consistent with the corresponding stage of the visible light backbone branch, thus forming a four-level geometric prior feature pyramid.

[0009] Furthermore, step S4, the process of de-entanglementing and fusing the visible light features and digital surface model features of each encoding stage, includes the following steps: S41. Apply global average pooling to the visible light features and digital surface model features respectively to obtain their respective global description vectors; multiply the two global description vectors channel by channel and normalize them with the Sigmoid function to obtain the feature similarity vector, which is used to measure the shared semantic response strength of each channel between the two modalities. S42. Multiply the visible light features and digital surface model features element-wise with the geometric position features of the corresponding stage to obtain the response representation of each mode to the three-dimensional spatial structure; calculate the absolute difference between the two response representations and take the mean in the spatial dimension to obtain the position response difference vector, which is used to measure the degree of dependence of each channel on modal-specific information. S43. After concatenating the feature similarity vector and the position response difference vector along the channel dimension, input them into the channel attention module consisting of two fully connected layers. The middle layer uses the ReLU activation function and the end layer uses the Sigmoid function for normalization to generate adaptive fusion weights. The fusion weights are used to dynamically weight and fuse the visible light features and the digital surface model features to obtain the fusion features at this stage.

[0010] Furthermore, in step S5, the deep feature fusion module includes three progressive stages: S51. The first stage is the selective deentanglement stage: it consists of two structurally symmetrical branches. After mapping the input fused features through a linear projection layer and a depthwise separable convolutional layer, some channels are crossed to construct a cross-reconstruction representation containing modality-specific attributes. Subsequently, the cross-reconstruction representation is rearranged into sequences along four directions: top left to bottom right, bottom right to top left, top right to bottom left, and bottom left to top right. Multi-directional contextual information is extracted and aggregated using four independent state space model branches. Combined with residual connections to maintain feature consistency, the output is a bimodal feature enhanced by preliminary deentanglement. S52. The second stage is the cross-modal mutual enhancement stage: the bimodal features processed in the first stage are input into multiple cascaded mutual enhancement modules; in each mutual enhancement module, the visible light branch and the digital surface model branch generate the system matrix of the state space model from their respective input features through linear mapping, but the output matrix is ​​generated by the other modality, so that each branch incorporates the information of the other modality when reconstructing the output from the hidden state. The state update and output reconstruction process is as follows: ; ; ; ; in, Let A be the hidden state of the DSM branch at time t. dsm The system matrix for the branch of the digital surface model, B dsm This is the input matrix for the digital surface model branch. and C represents the visible light and digital surface model inputs at time t, respectively. vis The output matrix of the visible light branch, C dsmThe output matrix of the digital surface model branch. For DSM branch output, Let A be the hidden state of the visible light branch at time t. vis For the visible light branching system matrix, B vis This is the input matrix for the visible light branch. For visible light branch output, D dsm DSM outputs the bias matrix, D vis This is the visible light output bias matrix; S53. The third stage is the bidirectional fusion stage: the dual-modal features enhanced in the first two stages are mapped through a linear projection layer and a depthwise separable convolutional layer, respectively, and then concatenated along the sequence dimension to construct a forward scanning sequence; the forward scanning sequence is reversed to obtain a reverse scanning sequence; the forward and reverse sequences are respectively input into independent state space models for processing, the processed sequences are flipped and accumulated to the original sequence and then concatenated, and compressed by linear projection to obtain deep fusion features.

[0011] Furthermore, in step S6, the operation of the cascaded decoder includes the following steps: S61. Use the reconstruction module to reconstruct the two-dimensional sequence output by the deep feature fusion module into a three-dimensional tensor; S62. The spatial resolution of the feature map is gradually restored to the size of the input image through multiple cascaded decoding blocks. Each decoding block upsamples the output of the previous stage and concatenates it with the fused features from the corresponding encoding stage along the channel dimension, and then processes it through a convolutional layer and a ReLU activation function in sequence. S63. Set up a segmentation head and a boundary detection auxiliary branch in parallel at the end of the decoder. The segmentation head outputs pixel-level semantic prediction, and the boundary detection auxiliary branch outputs boundary prediction. S64. The total loss function is defined as the weighted sum of semantic segmentation loss and boundary detection loss, where semantic segmentation loss is the weighted sum of pixel-level multi-class cross-entropy loss and Dice loss, and boundary detection loss adopts the form of binary cross-entropy.

[0012] Furthermore, in step S1, both the visible light backbone branch and the digital surface model backbone branch use ResNet as the backbone network, each consisting of four residual stages, and the two backbone branches are strictly aligned in the spatial scale of the output at each stage.

[0013] Furthermore, the geometric position prior module in step S3 is independent of the two main branches and uses a lightweight four-stage convolutional encoder to encode the three-dimensional coordinate tensor constructed from the digital surface model. The spatial scale of the output of each stage is strictly aligned with the main branches.

[0014] The beneficial effects of adopting this technical solution are: This invention explicitly encodes the three-dimensional geometric structure information contained in the digital surface model into geometric position features that can be aligned with optical features through a geometric position prior module, thus overcoming the shortcomings of existing methods in making insufficient use of the geometric information of the digital surface model. This invention uses a deentanglement fusion module to quantify the response differences of two modes to a three-dimensional spatial structure using geometric priors, thereby achieving adaptive separation and fusion of shared features and modality-specific features. This overcomes the problems of strong modal interference and insufficient detail preservation in traditional shallow fusion methods. This invention organically combines deentanglement, cross-modal mutual enhancement, and bidirectional scanning through a three-stage deep feature fusion module based on a state-space model. It achieves efficient global modeling of multimodal deep features with linear computational complexity while preserving two-dimensional spatial dependencies. This invention constructs a three-branch hierarchical multimodal fusion framework with symmetric dual-backbone structure and geometric priors, which systematically alleviates the problems of imbalance between local and global modeling and coarse cross-modal fusion in multimodal remote sensing segmentation. This invention introduces a multi-task learning strategy to jointly optimize semantic segmentation and boundary detection tasks, effectively improving the segmentation accuracy of complex ground feature boundaries and the segmentation consistency of ground features at multiple scales. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a multimodal remote sensing image semantic segmentation method based on geometric prior guidance in an embodiment of the present invention. Figure 2 This is a schematic diagram of the overall architecture of the segmentation network provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the de-entanglement fusion module provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings.

[0017] In this embodiment, see Figure 1 As shown, this invention proposes a multimodal remote sensing image semantic segmentation method based on geometric prior guidance, including the following steps: S1. After normalizing the remote sensing image to be segmented and its corresponding digital surface model data, slice it into multiple image block pairs using a sliding window method. S2. Input each image patch into the constructed symmetric dual-branch convolutional neural network backbone encoder. Both the visible light backbone branch and the digital surface model backbone branch use ResNet as the backbone network to extract multi-scale features of the visible light mode and the digital surface model mode respectively. The two branches are symmetrical and each contains four levels, and output shallow features of each scale respectively. S3. Using the geometric position prior module, pixel-level three-dimensional coordinates are constructed from the digital surface model data and multi-scale three-dimensional geometric position features are extracted. The geometric position prior module constitutes a third coding branch independent of the symmetrical dual-branch backbone encoder. S4. In each encoding stage, guided by the geometric position features, the response differences between visible light features and digital surface model features to the three-dimensional spatial structure are quantified, and the shared features and modality-specific features are adaptively de-entangled and weighted fused to generate multi-scale fused features. S5. Flatten the deepest fusion feature in the multi-scale fusion features into a sequence and input it into the deep feature fusion module. Based on the state space model, the extraction and fusion of multi-modal deep global context features are completed through three progressive stages: selective disentanglement, cross-modal mutual enhancement, and bidirectional fusion. S6. Input the fused features output by the deep feature fusion module into the cascade decoder, restore the spatial resolution layer by layer through skip connections, jointly optimize the semantic segmentation and boundary detection tasks, and output the final semantic segmentation result.

[0018] The remote sensing image to be segmented and its corresponding digital surface model data are input into a trained segmentation network to obtain a segmented image of the remote sensing image to be segmented. The segmentation network adopts an encoder-decoder structure. The encoder consists of a symmetrical visible light backbone branch, a digital surface model backbone branch, and a geometric branch, forming a three-branch encoder. An unentanglement fusion module is introduced at each scale layer, and a deep feature fusion module is introduced at the deepest layer. The decoder recovers the spatial resolution layer by layer through skip connections.

[0019] In a specific embodiment: such as Figure 2 As shown, the remote sensing image to be segmented and its corresponding digital surface model data are input into a trained segmentation network to obtain a segmented image of the remote sensing image to be segmented. Specifically, this includes the following steps: The remote sensing image to be segmented is normalized to the [0,1] interval along the channel direction to obtain the visible light modal image I. opt The digital surface model data is normalized to the [0,1] interval according to its elevation range to obtain the digital surface model image I. dsm ; Will I opt Input the visible light backbone branch to obtain the first-stage visible light feature f. vis,1 ; will f vis,1 The second-stage processing of the visible light backbone branch yields the second-stage visible light feature f. vis,2 ; will f vis,2 After the third stage of processing of the visible light backbone branch, the third stage visible light feature f is obtained. vis,3 ; will f vis,3After the fourth stage of processing in the visible light backbone branch, the fourth stage visible light feature f is obtained. vis,4 ; Will I dsm After being copied along the channel dimension to form a three-channel model, it is input into the main branch of the digital surface model to obtain the first-stage digital surface model features f. dsm,1 ; will f dsm,1 After the second stage of processing in the main branch of the digital surface model, the second-stage digital surface model features f are obtained. dsm,2 ; will f dsm,2 After the third stage of processing in the main branch of the digital surface model, the third-stage digital surface model features f are obtained. dsm,3 ; will f dsm,3 After the fourth stage of processing in the main branch of the digital surface model, the fourth stage digital surface model features f are obtained. dsm,4 ; Will I dsm Input the geometric location prior module to obtain the four-stage geometric prior features z. pos,1 z pos,2 z pos,3 z pos,4 ; f vis,1 f dsm,1 z pos,1 The input is processed by the de-entanglement fusion module to obtain the first fused feature M1; f vis,2 f dsm,2 z pos,2 The input is processed by the de-entanglement fusion module to obtain the second fused feature M2; f vis,3 f dsm,3 z pos,3 The input is processed by the de-entanglement fusion module to obtain the third fusion feature M3; f vis,4 f dsm,4 z pos,4 The input is processed by the de-entanglement fusion module to obtain the fourth fusion feature M4; After flattening M4 into a sequence, it is input into the deep feature fusion module for processing to obtain the deep fused feature F. fuse ; F fuse The reconstruction module of the input cascaded decoder is used to obtain the three-dimensional tensor D4; After upsampling D4, the third decoding feature D3 is obtained and concatenated with M3 along the channel dimension. Then, it is processed by a convolutional layer and a ReLU activation function to obtain the second decoding feature D2. After upsampling D2, it is concatenated with M2 along the channel dimension and processed by a convolutional layer and a ReLU activation function to obtain the first decoding feature D1. After upsampling D1, it is concatenated with M1 along the channel dimension and processed by a convolutional layer and a ReLU activation function. Simultaneously, it is upsampled to the size of the input image to obtain the final decoding feature D0. Input D0 into the segmentation head to obtain the semantic segmentation probability map P; input D0 into the boundary detection auxiliary branch to obtain the boundary prediction probability map Q; take the category with the highest probability along the category dimension of P to obtain the segmentation image Y of the remote sensing image to be segmented.

[0020] As an optimization of the above embodiment, both the visible light backbone branch and the digital surface model backbone branch use ResNet as the backbone network. Both have symmetrical structures, each consisting of four residual stages. Each residual stage comprises 1×1 convolution, 3×3 convolution, 1×1 convolution, and an identity mapping. The total downsampling factor of the entire backbone branch is 32. The spatial resolutions of the outputs of each stage of the two backbone branches are H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, respectively.

[0021] As an optimized solution of the above embodiments, the working process of the geometric position prior module includes the following steps: S31. For the pixel located in the i-th row and j-th column of the digital surface model image, construct a three-dimensional coordinate system with coordinates (x, y, j). i,j ,y i,j ,h i,j ), where x i,j With y i,j These represent the normalized row and column positions of the pixel in the two-dimensional plane, respectively, h. i,j This is the normalized elevation value at this pixel, with all three components mapped to the [0,1] interval; specifically: , , ; S32. Stack the three-dimensional coordinates of all pixels in spatial order to construct a three-dimensional coordinate tensor. The dimensions are H×W×3, where H and W are the height and width of the input image, respectively. S33. The three-dimensional coordinate tensor is fed into a four-stage hierarchical convolutional encoder. Each stage uses a convolutional layer with a kernel size of 3×3, a stride of 2, and padding of 1 for downsampling, followed by a ReLU activation function to progressively extract multi-scale geometric position features; The spatial resolution of the geometric position features output by each stage is strictly consistent with the corresponding stage of the visible light backbone branch, thus forming a four-level geometric prior feature pyramid.

[0022] Specifically: Will I pos The geometric prior features z are obtained by sequentially processing the first-stage 3×3 convolutional layer (stride 2, padding 1) and ReLU activation function. pos,1 ; z pos,1 The second-stage geometric prior features z are obtained by sequentially processing the data through a second-stage 3×3 convolutional layer (stride 2, padding 1) and a ReLU activation function. pos,2 ; z pos,2 The geometric prior features z are obtained by sequentially processing the data through a third-stage 3×3 convolutional layer (stride 2, padding 1) and a ReLU activation function. pos,3 ; z pos,3 The geometric prior features z are obtained by sequentially processing the data through a fourth-stage 3×3 convolutional layer (stride 2, padding 1) and a ReLU activation function. pos,4 ; in, Its spatial resolution The spatial resolution is strictly consistent with that of the corresponding stage of the visible light backbone branch.

[0023] As an optimization of the above embodiments, the process of de-entanglementing and fusing the visible light features and digital surface model features at each encoding stage includes the following steps: S41. Apply global average pooling to the visible light features and digital surface model features respectively to obtain their respective global description vectors; multiply the two global description vectors channel by channel and normalize them with the Sigmoid function to obtain the feature similarity vector, which is used to measure the shared semantic response strength of each channel between the two modalities. S42. Multiply the visible light features and digital surface model features element-wise with the geometric position features of the corresponding stage to obtain the response representation of each mode to the three-dimensional spatial structure; calculate the absolute difference between the two response representations and take the mean in the spatial dimension to obtain the position response difference vector, which is used to measure the degree of dependence of each channel on modal-specific information. S43. After concatenating the feature similarity vector and the position response difference vector along the channel dimension, input them into the channel attention module consisting of two fully connected layers. The middle layer uses the ReLU activation function and the end layer uses the Sigmoid function for normalization to generate adaptive fusion weights. The fusion weights are used to dynamically weight and fuse the visible light features and the digital surface model features to obtain the fusion features at this stage.

[0024] Specifically: like Figure 3 As shown, f vis,n f dsm,n z pos,n The input is processed by the de-entanglement fusion module to obtain the nth fused feature M. n (n=1,2,3,4), specifically including the following steps: f vis,n The visible light global description vector is obtained after processing by a global average pooling layer. ; f dsm,n The global average pooling layer is used to obtain the global description vector of the digital surface model. ; Will and Perform channel-by-channel multiplication to obtain the channel product vector; The channel product vector is processed by the Sigmoid activation function to obtain the feature similarity vector r. n ; f vis,n With z pos,n Element-wise multiplication is performed to obtain the response characteristics of visible light to the geometric structure. ; f dsm,n With z pos,n Element-wise multiplication is performed to obtain the response characteristics of the digital surface model to the geometric structure. ; Will and The absolute difference is obtained by taking the difference and then taking the absolute value. The mean of the absolute difference features is calculated in the spatial dimension to obtain the location response difference vector d. i ; r n With d n By concatenating along the channel dimension, we obtain the concatenated vector u. n ; will u n The mixture is processed sequentially through a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function to obtain the fusion weight w. n ; w n with f vis,n Perform element-wise multiplication to obtain the weighted visible light feature components; (1-w) n ) and f dsm,n Perform element-wise multiplication to obtain the weighted digital surface model feature components; The weighted visible light feature components are added element-wise to the weighted digital surface model feature components to obtain the nth fused feature M. n .

[0025] As an optimization of the above embodiment, step S5 includes three progressive stages in the deep feature fusion module: S51. The first stage is the selective deentanglement stage: it consists of two structurally symmetrical branches. After mapping the input fused features through a linear projection layer and a depthwise separable convolutional layer, some channels are crossed to construct a cross-reconstruction representation containing modality-specific attributes. Subsequently, the cross-reconstruction representation is rearranged into sequences along four directions: top left to bottom right, bottom right to top left, top right to bottom left, and bottom left to top right. Multi-directional contextual information is extracted and aggregated using four independent state space model branches. Combined with residual connections to maintain feature consistency, the output is a bimodal feature enhanced by preliminary deentanglement. S52. The second stage is the cross-modal mutual enhancement stage: the bimodal features processed in the first stage are input into multiple cascaded mutual enhancement modules; in each mutual enhancement module, the visible light branch and the digital surface model branch generate the system matrix of the state space model from their respective input features through linear mapping, but the output matrix is ​​generated by the other modality, so that each branch incorporates the information of the other modality when reconstructing the output from the hidden state. The state update and output reconstruction process is as follows: ; ; ; ; in, Let A be the hidden state of the DSM branch at time t. dsm The system matrix for the branch of the digital surface model, B dsm This is the input matrix for the digital surface model branch. and C represents the visible light and digital surface model inputs at time t, respectively. vis The output matrix of the visible light branch, C dsm The output matrix of the digital surface model branch. For DSM branch output, Let A be the hidden state of the visible light branch at time t. vis For the visible light branching system matrix, B vis This is the input matrix for the visible light branch. For visible light branch output, D dsm DSM outputs the bias matrix, D vis This is the visible light output bias matrix; S53. The third stage is the bidirectional fusion stage: the dual-modal features enhanced in the first two stages are mapped through a linear projection layer and a depthwise separable convolutional layer, respectively, and then concatenated along the sequence dimension to construct a forward scanning sequence; the forward scanning sequence is reversed to obtain a reverse scanning sequence; the forward and reverse sequences are respectively input into independent state space models for processing, the processed sequences are flipped and accumulated to the original sequence and then concatenated, and compressed by linear projection to obtain deep fusion features.

[0026] Specifically: After flattening M4 into a sequence, it is input into the deep feature fusion module for processing to obtain the deep fused feature F. fuse Specifically, it includes the following steps: Flatten M4 along the spatial dimensions to obtain the input sequence S. in ; will S in The channel is divided into a visible light correlation component x0 and a digital surface model correlation component y0 along the channel direction; x0 and y0 are processed by a selective deentanglement module to obtain deentangled and enhanced bimodal features x1 and y1. x1 and y1 are then processed by N2 cascaded cross-modal mutual enhancement modules to obtain cross-modal mutual enhancement bimodal features x2 and y2, where N2 is a positive integer. x2 and y2 are further enhanced by N3 cascaded selective deentanglement modules to obtain the final bimodal features. and Where N3 is a positive integer; Will and The input is processed by the bidirectional fusion module to obtain the deep fusion feature F. fuse .

[0027] Specifically, the input visible light correlation component x in y-related components of digital surface model in The input is processed by a selective deentanglement module to obtain the enhanced bimodal feature x. out With y out Specifically, it includes the following steps: x in The visible light mapping feature x' is obtained by sequentially processing through a linear projection layer and a depthwise separable convolutional layer; y in The data is processed sequentially through a linear projection layer and a depthwise separable convolutional layer to obtain the digital surface model mapping feature y'; then, some channels of x' and y' are crossed to obtain a cross-reconstructed representation containing modality-specific attributes. The cross-reconstruction representation is selectively scanned along the upper left to lower right direction to obtain a first-direction scan result; the cross-reconstruction representation is selectively scanned along the lower right to upper left direction to obtain a second-direction scan result; the cross-reconstruction representation is selectively scanned along the upper right to lower left direction to obtain a third-direction scan result; the cross-reconstruction representation is selectively scanned along the lower left to upper right direction to obtain a fourth-direction scan result; the first to fourth-direction scan results are added and aggregated to obtain multi-directional contextual features; Combine the multi-directional context features with x in By summing the residuals, the enhanced visible light component x is obtained. out ; Combine the multi-directional context features with y in By summing the residuals, the enhanced digital surface model component y is obtained. out .

[0028] Specifically: The bimodal feature x in With y in The input is processed by the cross-modal mutual enhancement module to obtain the mutually enhanced bimodal features x. out With y out Specifically, it includes the following steps: x in The system matrix A of the visible light branch is obtained through linear mapping. vis Input matrix B vis Output matrix C vis With discretization step size ∆ vis ; will y in Through linear mapping, the system matrix A of the digital surface model branch is obtained. dsm Input matrix B dsm Output matrix C dsm With discretization step size ∆ dsm ; For A vis B vis According to step size ∆ vis Discretize the state space to obtain the discretized system matrix; for A dsm B dsm According to step size ∆ dsm Discretize the state space to obtain the discretized system matrix; For each time step t of the visible light branch, according to Update the visible light hidden state; use the output matrix C generated by the digital surface model branch. dsm ,according to Reconstruct the output of the visible light branch; For each time step t of the digital surface model branch, according to Update the hidden state of the digital surface model; use the visible light branch to generate the output matrix C.vis ,according to The output of the reconstructed digital surface model branch; The visible light output sequence of all time steps is used as the mutually enhanced visible light component x. out ; The output sequence of the digital surface model at all time steps is used as the mutually enhanced digital surface model component y. out .

[0029] Specifically, dual-modal features and The input is processed by the bidirectional fusion module to obtain the deep fusion feature F. fuse Specifically, it includes the following steps: Will The visible light features are obtained by sequentially processing the data through a linear projection layer and a depthwise separable convolutional layer. ;Will The digital surface model features are obtained by sequentially processing the data through a linear projection layer and a depthwise separable convolutional layer. ; Will and By concatenating along the sequence dimensions, a forward scan sequence S is obtained. Forward ; For S Forward Perform a reverse rearrangement to obtain the reverse scan sequence S. Inverse ; S Forward The input state-space model is processed to obtain a positive output sequence; S Inverse The input state-space model is processed to obtain the inverse output sequence; After flipping the positive output sequence, it is compared with S. Forward Add them together to obtain the forward fusion result; flip the reverse output sequence and then combine it with S. Inverse Add them together to obtain the reverse fusion result; The forward fusion result and the reverse fusion result are concatenated along the channel dimension to obtain preliminary fusion features; the preliminary fusion features are then compressed to the target number of channels using linear projection to obtain deep fusion features F. fuse .

[0030] As an optimization of the above embodiment, step S6, the operation of the cascaded decoder includes the following steps: S61. Use the reconstruction module to reconstruct the two-dimensional sequence output by the deep feature fusion module into a three-dimensional tensor; S62. The spatial resolution of the feature map is gradually restored to the size of the input image through multiple cascaded decoding blocks. Each decoding block upsamples the output of the previous stage and concatenates it with the fused features from the corresponding encoding stage along the channel dimension, and then processes it through a convolutional layer and a ReLU activation function in sequence. S63. Set up a segmentation head and a boundary detection auxiliary branch in parallel at the end of the decoder. The segmentation head outputs pixel-level semantic prediction, and the boundary detection auxiliary branch outputs boundary prediction. S64. The total loss function is defined as the weighted sum of semantic segmentation loss and boundary detection loss, where semantic segmentation loss is the weighted sum of pixel-level multi-class cross-entropy loss and Dice loss, and boundary detection loss adopts the form of binary cross-entropy.

[0031] Specifically: Deeply fused features F fuse The multi-scale fused features M1, M2, and M3 are input into a cascaded decoder for processing to obtain a semantic segmentation image Y. This process includes the following steps: F fuse Input the reconstruction module to obtain the initial 3D tensor: ; D4 is upsampled by a factor of 2 to obtain the upsampled feature U3; U3 and M3 are concatenated along the channel dimension to obtain the third concatenated feature; the third concatenated feature is then processed sequentially through a 3×3 convolutional layer and a ReLU activation function to obtain the third decoding feature D3; D3 is upsampled by 2 times to obtain the upsampled feature U2; U2 and M2 are concatenated along the channel dimension to obtain the second concatenated feature; the second concatenated feature is processed sequentially through a 3×3 convolutional layer and a ReLU activation function to obtain the second decoding feature D2; D2 is upsampled by 2 times to obtain the upsampled feature U1; U1 and M1 are concatenated along the channel dimension to obtain the first concatenated feature; the first concatenated feature is processed sequentially through a 3×3 convolutional layer and a ReLU activation function to obtain the first decoding feature D1; D1 is upsampled by 4 times to the input image size H×W to obtain the final decoding feature D0; D0 is processed sequentially through a 1×1 convolutional layer and a Softmax activation function to obtain the semantic segmentation probability map P∈R. H×W×K Where K is the number of land cover categories; D0 is processed sequentially through a 1×1 convolutional layer and a Sigmoid activation function to obtain the boundary prediction probability map Q∈R. H×W×1 ; For P, take the category with the highest probability along the category dimension to obtain the segmentation image Y(i,j)=argmax of the remote sensing image to be segmented. k P k (i,j).

[0032] Specifically, the segmentation network is trained based on the following total loss function: L total =L seg +λ·Lboundary ; Among them, L seg L represents the semantic segmentation loss. boundary Let λ represent the boundary detection loss, and λ be the balance coefficient.

[0033] Specifically, the semantic segmentation loss L is obtained based on the following formula. seg With boundary detection loss L boundary : ; ; ; ; In the formula, N represents the number of pixels in the remote sensing image, and K represents the number of land cover categories; y i,k This represents the true label of the i-th pixel. If the i-th pixel belongs to the k-th land cover category, then y... i,k It equals 1 if it is not equal to 0 otherwise; p i,k b represents the probability value of predicting that the i-th pixel belongs to the k-th land cover category; i The true boundary label of the i-th pixel is obtained by performing a morphological gradient operation on the feature label map; q i denoted as the boundary prediction probability value of the i-th pixel; α, β, and λ are balance coefficients.

[0034] The above formulas are all dimensionless calculations. The preset parameters, weights and thresholds in the formulas shall be set by those skilled in the art according to the actual situation.

[0035] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A semantic segmentation method for multimodal remote sensing images based on geometric prior guidance, characterized in that, Including the following steps: S1. After normalizing the remote sensing image to be segmented and its corresponding digital surface model data, slice it into multiple image block pairs using a sliding window method. S2. Input each image patch into the constructed symmetric dual-branch convolutional neural network backbone encoder. Both the visible light backbone branch and the digital surface model backbone branch use ResNet as the backbone network to extract multi-scale features of the visible light mode and the digital surface model mode respectively. The two branches are symmetrical and each contains four levels, and output shallow features of each scale respectively. S3. Using the geometric position prior module, pixel-level three-dimensional coordinates are constructed from the digital surface model data and multi-scale three-dimensional geometric position features are extracted. The geometric position prior module constitutes a third coding branch independent of the symmetrical dual-branch backbone encoder. S4. In each encoding stage, guided by the geometric position features, the response differences between visible light features and digital surface model features to the three-dimensional spatial structure are quantified, and the shared features and modality-specific features are adaptively de-entangled and weighted fused to generate multi-scale fused features. S5. Flatten the deepest fusion feature in the multi-scale fusion features into a sequence and input it into the deep feature fusion module. Based on the state space model, the extraction and fusion of multi-modal deep global context features are completed through three progressive stages: selective disentanglement, cross-modal mutual enhancement, and bidirectional fusion. S6. Input the fused features output by the deep feature fusion module into the cascade decoder, restore the spatial resolution layer by layer through skip connections, jointly optimize the semantic segmentation and boundary detection tasks, and output the final semantic segmentation result.

2. The multimodal remote sensing image semantic segmentation method based on geometric prior guidance according to claim 1, characterized in that, In step S3, the working process of the geometric position prior module includes the following steps: S31. For the pixel located in the i-th row and j-th column of the digital surface model image, construct a three-dimensional coordinate system with coordinates (x, y, j). i,j ,y i,j ,h i,j ), where x i,j With y i,j These represent the normalized row and column positions of the pixel in the two-dimensional plane, respectively, h. i,j This is the normalized elevation value at this pixel, with all three components mapped to the [0,1] interval; S32. Stack the three-dimensional coordinates of all pixels in the image in spatial order to construct a three-dimensional coordinate tensor with a size of H×W×3, where H and W are the height and width of the input image, respectively. S33. The three-dimensional coordinate tensor is fed into a four-stage hierarchical convolutional encoder. Each stage uses a convolutional layer with a kernel size of 3×3, a stride of 2, and padding of 1 for downsampling, followed by a ReLU activation function to progressively extract multi-scale geometric position features; The spatial resolution of the geometric position features output by each stage is strictly consistent with the corresponding stage of the visible light backbone branch, thus forming a four-level geometric prior feature pyramid.

3. The multimodal remote sensing image semantic segmentation method based on geometric prior guidance according to claim 1, characterized in that, In step S4, the process of de-entanglementing and fusing the visible light features and digital surface model features at each encoding stage includes the following steps: S41. Apply global average pooling to the visible light features and digital surface model features respectively to obtain their respective global description vectors; multiply the two global description vectors channel by channel and normalize them with the Sigmoid function to obtain the feature similarity vector, which is used to measure the shared semantic response strength of each channel between the two modalities. S42. Multiply the visible light features and digital surface model features element-wise with the geometric position features of the corresponding stage to obtain the response representation of each mode to the three-dimensional spatial structure; calculate the absolute difference between the two response representations and take the mean in the spatial dimension to obtain the position response difference vector, which is used to measure the degree of dependence of each channel on modal-specific information. S43. After concatenating the feature similarity vector and the position response difference vector along the channel dimension, input them into the channel attention module consisting of two fully connected layers. The middle layer uses the ReLU activation function and the end layer uses the Sigmoid function for normalization to generate adaptive fusion weights. The fusion weights are used to dynamically weight and fuse the visible light features and the digital surface model features to obtain the fusion features at this stage.

4. The multimodal remote sensing image semantic segmentation method based on geometric prior guidance according to claim 1, characterized in that, In step S5, the deep feature fusion module includes three progressive stages: S51. The first stage is the selective deentanglement stage: it consists of two structurally symmetrical branches. After mapping the input fused features through a linear projection layer and a depthwise separable convolutional layer, some channels are crossed to construct a cross-reconstruction representation containing modality-specific attributes. Then, the cross-reconstruction representation is rearranged into a sequence along four directions: top left to bottom right, bottom right to top left, top right to bottom left, and bottom left to top right. Multi-directional contextual information is extracted and aggregated using four independent state space model branches. By combining residual connections to maintain feature consistency, the output is a bimodal feature enhanced by preliminary deentanglement. S52. The second stage is the cross-modal mutual enhancement stage: the dual-modal features processed in the first stage are input into multiple cascaded mutual enhancement modules; In each mutual enhancement module, the visible light branch and the digital surface model branch generate the system matrix of the state space model from their respective input features through linear mapping, but the output matrix is ​​generated by the other modality. This allows each branch to incorporate information from the other modality when reconstructing the output from the hidden state. The state update and output reconstruction process is as follows: ; ; ; ; in, Let A be the hidden state of the DSM branch at time t. dsm The system matrix for the branch of the digital surface model, B dsm This is the input matrix for the digital surface model branch. and C represents the visible light and digital surface model inputs at time t, respectively. vis The output matrix of the visible light branch, C dsm The output matrix of the digital surface model branch. For DSM branch output, Let A be the hidden state of the visible light branch at time t. vis For the visible light branching system matrix, B vis This is the input matrix for the visible light branch. For visible light branch output, D dsm DSM outputs the bias matrix, D vis This is the visible light output bias matrix; S53. The third stage is the bidirectional fusion stage: the dual-modal features enhanced in the first two stages are mapped through a linear projection layer and a depthwise separable convolutional layer, respectively, and then concatenated along the sequence dimension to construct a forward scanning sequence; the forward scanning sequence is reversed to obtain a reverse scanning sequence; the forward and reverse sequences are respectively input into independent state space models for processing, the processed sequences are flipped and accumulated to the original sequence and then concatenated, and compressed by linear projection to obtain deep fusion features.

5. The multimodal remote sensing image semantic segmentation method based on geometric prior guidance according to claim 1, characterized in that, In step S6, the operation of the cascaded decoder includes the following steps: S61. Use the reconstruction module to reconstruct the two-dimensional sequence output by the deep feature fusion module into a three-dimensional tensor; S62. The spatial resolution of the feature map is gradually restored to the size of the input image through multiple cascaded decoding blocks. Each decoding block upsamples the output of the previous stage and concatenates it with the fused features from the corresponding encoding stage along the channel dimension, and then processes it through a convolutional layer and a ReLU activation function in sequence. S63. Set up a segmentation head and a boundary detection auxiliary branch in parallel at the end of the decoder. The segmentation head outputs pixel-level semantic prediction, and the boundary detection auxiliary branch outputs boundary prediction. S64. The total loss function is defined as the weighted sum of semantic segmentation loss and boundary detection loss, where semantic segmentation loss is the weighted sum of pixel-level multi-class cross-entropy loss and Dice loss, and boundary detection loss adopts the form of binary cross-entropy.

6. The multimodal remote sensing image semantic segmentation method based on geometric prior guidance according to claim 1, characterized in that, In step S1, both the visible light backbone branch and the digital surface model backbone branch use ResNet as the backbone network, each consisting of four residual stages, and the two backbone branches are strictly aligned in the spatial scale of the output of each stage.

7. The multimodal remote sensing image semantic segmentation method based on geometric prior guidance according to claim 6, characterized in that, The geometric position prior module in step S3 is independent of the two main branches. It uses a lightweight four-stage convolutional encoder to encode the three-dimensional coordinate tensor constructed from the digital surface model. The spatial scale of the output of each stage is strictly aligned with the main branches.

Images