A multi-level multi-modal remote sensing image semantic segmentation method and device

Abstract

This invention discloses a multi-level, multimodal remote sensing image semantic segmentation method and apparatus, comprising: performing dual-stream feature extraction and cross-modal joint context recalibration based on preprocessed bimodal image data to obtain cross-modal recalibration features; performing spatial adaptive low-pass upsampling and noise suppression based on the cross-modal recalibration features to obtain a high-quality upsampled feature map, and performing local consistency optimization and boundary semantic alignment to obtain a boundary-optimized feature map; acquiring skip connection features at different scales; performing multi-scale dual-pooling aggregation and multi-level feature fusion based on the boundary-optimized feature map and the skip connection features at different scales to obtain pixel-level semantic prediction results; obtaining the ground truth pixel label values ​​of the corresponding regions of ground features; and optimizing and training network parameters through a combined loss function based on the pixel-level semantic prediction results and the ground truth pixel label values ​​to obtain a final high-precision multimodal semantic classification map.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of remote sensing data processing, computer vision, and deep learning in artificial intelligence. Specifically, it relates to a method and apparatus for semantic segmentation of remote sensing images based on a multi-level, multimodal architecture. This technology is mainly applied to the accurate, pixel-level classification and extraction of complex land cover areas in high-resolution remote sensing images, aiming to provide reliable underlying algorithms and technical support for practical applications such as modern urban land cover change analysis. Background Technology

[0002] In the field of remote sensing data processing, semantic segmentation is a core technology for achieving accurate, pixel-level classification of land cover areas in high-resolution remote sensing images. Early multimodal data processing often relied on traditional feature extraction (e.g., vegetation index, spectral fusion analysis) or traditional machine learning classifiers. These methods typically require a significant amount of manual feature design, are difficult to adapt to the complex nonlinear mapping relationships between multi-source data, and are highly susceptible to interference from local atmospheric conditions, lighting, and other environmental factors, making them unsuitable for the massive and complex analytical needs of modern remote sensing images.

[0003] In recent years, deep learning-based multimodal feature fusion methods (e.g., combining RGB color images with Digital Surface Models (DSM)) have become the mainstream for semantic segmentation of remote sensing images. However, existing deep learning methods still have significant limitations: on the one hand, models based on classic convolutional neural networks (CNNs) are limited by their local receptive fields, making them struggle with global context modeling and prone to misclassification when dealing with complex terrain distribution areas (e.g., large areas of densely interspersed low vegetation and trees); on the other hand, although methods based on Transformers and hybrid architectures compensate for the lack of global modeling, existing networks typically employ shallow fusion or simple attention stitching, resulting in insufficient communication between different modal data and making it difficult for the model to distinguish easily confused features with extremely similar colors or heights (e.g., low vegetation and roads). Furthermore, in the image decoding and reconstruction stage, most existing models rely on traditional bilinear upsampling mechanisms. This coarse feature reconstruction often leads to the loss of high-frequency details and introduces a large amount of feature aliasing noise, making the predicted edges of complex objects overly smooth and blurry, and inadequate for handling fine-grained geometric boundaries.

[0004] Therefore, there is an urgent need for a semantic segmentation method that can fully tap the potential of cross-modal deep collaboration and effectively overcome the challenges of upsampling feature ambiguity and boundary alignment, so as to improve the classification accuracy and robustness of remote sensing systems in complex lighting and varied terrain environments. Summary of the Invention

[0005] This invention provides a multi-level, multi-modal remote sensing image semantic segmentation method and apparatus. This invention achieves high-precision, sharp boundary segmentation of complex features (such as low vegetation, buildings, and small vehicles) in high-resolution remote sensing images, effectively solving the problems of insufficient utilization of multi-modal data and noise interference caused by upsampling detail degradation. It provides reliable technical support for modern urban remote sensing monitoring and refined land cover analysis. See the description below for details:

[0006] Firstly, a multi-level, multi-modal remote sensing image semantic segmentation method, the method comprising:

[0007] The optical color image and digital surface model data of the region to be processed are acquired, and differential preprocessing and spatial enhancement are performed on the optical color image and digital surface model data to obtain preprocessed dual-modal image data.

[0008] Based on the preprocessed dual-modal image data, dual-stream feature extraction and cross-modal joint context recalibration are performed to obtain cross-modal recalibration features;

[0009] Based on the cross-modal recalibration features, spatial adaptive low-pass upsampling and noise suppression are performed to obtain high-quality upsampled feature maps. Local consistency optimization and boundary semantic alignment are then performed to obtain boundary-optimized feature maps.

[0010] By acquiring skip connection features at different scales, and based on the feature maps optimized by the boundary and the skip connection features at different scales, multi-scale dual pooling aggregation and multi-level feature fusion are performed to obtain pixel-level semantic prediction results.

[0011] The true pixel label values ​​of the corresponding regions of ground features are obtained. Based on the pixel-level semantic prediction results and the true pixel label values, the network parameters are optimized and trained by combining loss functions to obtain the final high-precision multimodal semantic classification map.

[0012] Specifically, based on the preprocessed dual-modal image data, dual-stream feature extraction and cross-modal joint context recalibration are performed to obtain the cross-modal recalibration features as follows:

[0013] In the feature encoding stage, a joint context-aware shallow feature fusion module is introduced. Through global average pooling, the joint global context information of the two-stream features is extracted, and cross-modal collaborative vectors are generated accordingly. Independent fully connected activation paths are used to generate collaborative weights for each branch. Channel-level multiplication recalibration is performed on the original features using the joint weights.

[0014] The step of performing spatial adaptive low-pass upsampling and noise suppression based on cross-modal recalibration characteristics is as follows:

[0015] By utilizing a spatially adaptive low-pass upsampling module, a lightweight convolutional network autonomously predicts low-pass filter weights corresponding to spatial variations in the local neighborhood. Sub-pixel convolutional reconstruction technology is applied to the normalized filter weights to dynamically attenuate high-frequency aliasing noise and feature aliasing interference during the upsampling of calibrated multimodal features.

[0016] The local consistency optimization and boundary semantic alignment are as follows:

[0017] The upsampled feature map is input into local consistency optimization, and the feature cosine similarity between each pixel and its 8 neighboring pixels is calculated to construct a local similarity matrix. Based on the initial fused features and similarity matrix, a lightweight convolutional network is used to predict the resampling offset direction and magnitude of each pixel in two-dimensional space, and deformable resampling operation is performed.

[0018] The cross-modal cooperation vector is:

[0019] Extracted RGB features and DSM characteristics Global average pooling is used to aggregate the global spatial information of the two streams. The aggregated features are then concatenated to generate a joint context vector for cross-modal cooperation. The calculation formula is as follows:

[0020] .

[0021] The calibrated multimodal features are as follows:

[0022]

[0023] Where X represents the low-resolution input feature. The output feature value at coordinate (i, j) on the target high-resolution feature map; The filtering weights are dynamically generated for the target position (i, j) and have a relative offset of (u, v) within the corresponding local neighborhood. The input is the feature value of the neighboring pixels corresponding to the target location on the low-resolution feature map; These are the horizontal and vertical spatial offset indices within the local filter kernel; It is the set of local neighborhood spaces covered by the dynamic filter kernel.

[0024] The multi-scale dual-pooling polymerization is as follows:

[0025] The upper branch uses max pooling to extract obvious local high-frequency textures; the lower branch uses average pooling to preserve globally smooth background information.

[0026] The two pooling results are concatenated and fused with the original features, and residual connections are used. The aggregation process is represented as follows:

[0027]

[0028] in, This is for the final multi-scale fusion feature representation; This represents the feature concatenation operation performed along the channel dimension; F is the input feature map to be processed.

[0029] In a second aspect, a remote sensing image semantic segmentation device based on multi-level multimodal processing is provided. The device includes a processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to perform the method described in any one of the first aspects.

[0030] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0031] The beneficial effects of the technical solution provided by this invention are:

[0032] 1. This invention proposes a multi-level, multimodal remote sensing image semantic segmentation framework called JAD-TransUNet. This framework adopts a symmetrical U-shaped structure, combining the ability of convolutional neural networks (CNN) to extract local features with the ability of Transformers to process global information. This enables the model to better coordinate the processing of optical information (RGB) and provide a highly prior digital surface model (DSM), effectively overcoming the limitations of single modality in the identification of complex ground features and significantly improving the classification accuracy of remote sensing images.

[0033] 2. The innovative cross-modal feature fusion mechanism of this invention: The Joint Context-Aware Shallow Feature Fusion (JC-SE) module designed in this invention breaks through the limitations of previous independent processing of each modality weight. This module generates collaborative weights by aggregating joint global context information, enabling cross-modal collaboration between color images and height data. This mechanism allows each feature to perceive the global state of another modality during enhancement, effectively solving the technical problem that the model has difficulty distinguishing between features with similar colors (e.g., grass and trees) or similar heights (e.g., low plants and roads).

[0034] 3. Spatial Adaptive High-Efficiency Decoding and Noise Suppression: To address the problem that traditional upsampling operations easily lead to the loss of high-frequency details and blurred edges, this invention constructs a spatial adaptive decoding network; among which, the Spatial Adaptive Low-Pass Upsampling (SALU) module can autonomously generate dynamic filters, which effectively preserve clear image edge information while smoothing internal image noise interference, thus alleviating the detail degradation caused by upsampling from the source.

[0035] 4. Local Consistency Boundary Alignment and Multi-Scale Enhancement: This invention introduces a Local Consistency Optimizer (LCO) and a Multi-Scale Dual Pooling (MSDP) module. LCO performs deformable resampling by calculating local feature similarity and predicting pixel-level offsets, accurately correcting boundary position deviations. The MSDP module combines max pooling and average pooling operations to extract local texture and global background information, significantly improving the model's comprehensive discrimination ability against large homogeneous regions and small, complex targets.

[0036] 5. This invention uses overall accuracy (OA), mean intersection-over-union ratio (mIoU), and F1 score as evaluation metrics to effectively evaluate the model. It achieves superior performance compared to current mainstream advanced methods (such as existing CNN-based and Transformer-based models) on publicly available remote sensing land cover datasets (ISPRS Vaihingen and ISPRS Potsdam). This solution verifies its high accuracy and reliability in real-world complex urban scenarios, providing precise technical support for land cover analysis and resource management in the field of remote sensing. Attached Figure Description

[0037] Figure 1 The overall network architecture diagram of JAD-TransUNet;

[0038] Figure 2 Structure diagram of the Joint Context-Aware Shallow Feature Fusion (JC-SE) module;

[0039] Figure 3 Detailed flowchart of the Spatial Adaptive Low-Pass Upsampling (SALU) module;

[0040] Figure 4 This is a diagram of the Local Consistency Optimizer (LCO) architecture.

[0041] Figure 5 This is a diagram of the Multi-Scale Dual Pooling (MSDP) module structure. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0043] In the decoding stage, this invention constructs a spatially adaptive network, which uses spatially adaptive low-pass upsampling (SALU) to autonomously generate dynamic filters to suppress high-frequency aliasing noise. It also combines a local consistency optimizer (LCO) to predict pixel-level offsets based on similarity and performs deformable resampling to accurately align semantic boundaries. Furthermore, this invention employs a multi-scale dual pooling (MSDP) module to comprehensively capture local details and global background. Finally, it combines a combined loss training strategy targeting easily confused features to achieve high-precision, sharp boundary segmentation of complex features (e.g., low vegetation, buildings, small vehicles) in high-resolution remote sensing images. This effectively solves the problems of insufficient utilization of multimodal data and noise interference caused by upsampling detail degradation, providing reliable technical support for modern urban remote sensing monitoring and refined land cover analysis.

[0044] Example 1

[0045] A multi-level, multimodal remote sensing image semantic segmentation method is proposed, comprising six core steps: multimodal data preprocessing, dual-stream coding and shallow feature fusion, spatial adaptive low-pass upsampling, local consistency optimization, multi-scale dual-pooling fusion, and combined loss training, as detailed below:

[0046] 101: Acquire optical color image (RGB) and digital surface model (DSM) data of the area to be processed, and perform differential preprocessing and spatial enhancement on the optical color image and digital surface model data to obtain preprocessed bimodal image data;

[0047] Specifically, the pixel values ​​of the dual-modal data are uniformly normalized and mapped to the [0, 1] distribution range to accelerate model convergence and preserve the original spectral and elevation information. Differentiated cutting logic is adopted for datasets of different sizes. For normalized datasets such as ISPRS Vaihingen, the original morphological input is preserved, while for ultra-large images such as ISPRS Potsdam, a sliding window algorithm is used to cut them into 1024 × 1024 pixel image blocks to adapt to hardware memory limitations and increase the number of training samples. At the same time, various spatial geometric data augmentation techniques such as random rotation, horizontal mirror flipping, and vertical flipping are introduced during the training phase to significantly enhance the model's generalization performance and robustness to features of different spatial orientations.

[0048] 102: Based on the preprocessed dual-modal image data, dual-stream feature extraction and cross-modal joint context recalibration are performed to obtain cross-modal recalibration features;

[0049] Specifically, the preprocessed RGB and DSM images are input into a two-stream encoder containing convolutional layers (such as ResNet50 pre-trained on ImageNet) and a Transformer module, respectively. The convolutional branches capture local fine-grained textures, and the Transformer module models long-range dependencies to capture global semantic information. In the feature encoding stage, a Joint Context-Aware Shallow Feature Fusion (JC-SE) module is introduced to aggregate the joint global contextual information of the two-stream features through a global average pooling (GAP) operation to generate cross-modal cooperative vectors. It also utilizes independent fully connected incentive paths to generate collaborative weights for each branch. and Finally, the joint weights are used to perform channel-level multiplication recalibration on the original features, so that each feature can perceive the global state of another modality in real time during enhancement, thereby effectively resolving the problem of misclassification of objects with extremely similar colors or heights.

[0050] 103: Based on the cross-modal recalibration features, spatial adaptive low-pass upsampling and noise suppression are performed to obtain high-quality upsampled feature maps;

[0051] Specifically, in the decoding stage of recovering high-resolution deep coarse features, the Spatial Adaptive Low-Pass Upsampling (SALU) module is used to replace the traditional bilinear interpolation mechanism. A lightweight convolutional network autonomously predicts the low-pass filter weights W corresponding to the spatial variations in the local neighborhood. Subsequently, sub-pixel convolution reconstruction technology is applied to the normalized filter weights. During the upsampling of the calibrated multimodal features, the high-frequency aliasing noise and feature aliasing interference introduced by the traditional interpolation operation are dynamically attenuated. This process alleviates the problem of detail degradation and boundary blurring in the image restoration stage from the source by selectively smoothing the internal features of the target and preserving the key edge structures, thus obtaining a high-quality upsampled feature map.

[0052] 104: Based on the high-quality upsampled feature map, perform local consistency optimization and boundary semantic alignment to obtain the boundary-optimized feature map;

[0053] Specifically, the upsampled feature map is input into the Local Consistency Optimizer (LCO). A local similarity matrix is ​​constructed by calculating the feature cosine similarity between each pixel and its 8 neighboring pixels, thereby accurately locating "candidate replacement pixels" with high similarity to enhance the consistency of local features. Based on the initial fused features and the similarity matrix, a lightweight convolutional network is used to predict the resampling offset direction and magnitude of each pixel in two-dimensional space, and deformable resampling operation is performed. This step replaces features in low similarity regions such as object edges or noise points with high similarity features in the neighborhood, forcing the feature positions to be precisely aligned with the true semantic boundaries, effectively correcting the boundary position deviation caused by receptive field limitations or upsampling.

[0054] 105: Obtain skip connection features at different scales. Based on the feature map after boundary optimization and skip connection features at different scales, perform multi-scale dual pooling aggregation and multi-level feature fusion to obtain pixel-level semantic prediction results.

[0055] Specifically, the Multi-Scale Dual Pooling (MSDP) module is used to simultaneously perform Max Pooling and Average Pooling operations on the boundary-optimized feature map and the multi-scale features from the skip connections. The Max Pooling branch is responsible for extracting significant local high-frequency textures (such as the right-angled outlines of buildings), while the Average Pooling branch is dedicated to preserving globally smooth background environmental information. By combining the residual connection mechanism and the cross-scale fusion strategy, the dual pooling features of different levels are spliced ​​and aggregated in the channel dimension, which effectively prevents the loss of feature information and gradient vanishing during the transmission process of deep networks. Through this multi-scale feature enhancement mechanism, the model's comprehensive discrimination ability for large homogeneous regions (such as water bodies) and small complex targets (such as vehicles) is significantly improved, and based on this, the final high-precision pixel-level semantic prediction results are generated.

[0056] 106: Obtain the true pixel label values ​​of the corresponding regions of ground features. Based on the pixel-level semantic prediction results and the true pixel label values, optimize the network parameters through a combination loss function to obtain the final high-precision multimodal semantic classification map.

[0057] Specifically, during the model training phase, a weighted cross-entropy loss function is constructed ( With Dice loss ( The combined loss function, composed of [missing information], collaboratively optimizes the accuracy of pixel-by-pixel classification and the segmentation quality of object edges. A class balancing strategy is introduced, applying higher penalty weights to easily confused categories such as low-lying vegetation or those with small sample sizes. The AdamW optimizer is employed, with an initial learning rate of 2 × [missing information]. It also employs a cosine annealing learning rate decay strategy to optimize the convergence process; and uses the backpropagation algorithm to iteratively update all learnable network parameters, including the dual-stream encoder, JC-SE, SALU, and LCO modules, on a computing server equipped with a high-performance graphics card (such as NVIDIA A40), ultimately outputting a pixel-level multimodal semantic classification map with high accuracy and sharp boundaries.

[0058] Specifically, the learnable network parameters include: the convolutional kernel weights in the dual-stream encoder and the attention mapping matrix of the Transformer module, the fully connected layer parameters used to generate collaborative weights in the JC-SE module, the convolutional parameters used to predict dynamic filters in the SALU module, and the lightweight network weights used to predict pixel-level offsets in the LCO module.

[0059] In summary, the embodiments of the present invention, by constructing a dual-stream coding architecture including a convolutional neural network (CNN) and a Transformer, and innovatively introducing a joint context-aware shallow feature fusion module (JC-SE), break through the limitations of traditional multimodal independent processing and realize cross-modal global context collaboration and dynamic feature recalibration of optical images (RGB) and elevation data (DSM).

[0060] Example 2

[0061] The solution in Embodiment 1 will be further described below with reference to specific technical details:

[0062] 201: Acquire remote sensing image data and perform basic preprocessing;

[0063] like Figure 1 The network architecture diagram shown depicts, from left to right, the network input receiving multimodal data. First, the pixel values ​​of the input optical color image (RGB) and digital surface model (DSM) are uniformly normalized and mapped to the [0,1] distribution range to accelerate model convergence. Simultaneously, during the training phase, data augmentation techniques such as random rotation, horizontal and vertical flipping are introduced to enhance the model's robustness to different spatial orientation features.

[0064] 202: The JC-SE (Joint Context-Aware Shallow Feature Fusion) module extracts the joint context and generates collaborative weights;

[0065] like Figure 2 The JC-SE module structure diagram shown below, viewed from left to right and top to bottom: First, the extracted RGB features... and DSM characteristics Global Average Pooling (GAP) is used to aggregate the global spatial information of the two streams. The aggregated features are then concatenated to generate a joint context vector for cross-modal collaboration. The calculation formula is as follows:

[0066]

[0067] then, The inputs are fed into two independent activation paths consisting of fully connected (FC) layers. The nonlinear relationships between channels are learned through dimensionality reduction and dimensionality increase operations, and finally, collaborative weights for the RGB and DSM branches are generated using the Sigmoid activation function, respectively. and Taking the RGB branch as an example, its weight is calculated as follows:

[0068]

[0069]

[0070] Among them, W1, W2, and For the weights of the fully connected layer, This is the ReLU activation function. This represents the Sigmoid activation function; This is the joint context vector for cross-modal collaboration. Finally, the joint weights are used to perform channel-level multiplication recalibration with the original features to obtain enhanced features, enabling each feature path to fully perceive the global state of the other modality.

[0071] 203: The SALU (Spatial Adaptive Low-Pass Upsampling) module performs sub-pixel feature reconstruction;

[0072] like Figure 3 The SALU module structure diagram shown abandons the traditional bilinear upsampling mechanism. From left to right, the input is a low-resolution deep feature map. The module first predicts the spatial variation filter weights W corresponding to the local neighborhood at coarse-grained locations through a lightweight convolutional network. Subsequently, sub-pixel convolution is applied to the normalized filter weights. For any location (i, j) on the target high-resolution feature map, its eigenvalue is calculated using the following formula:

[0073]

[0074] Where X represents the low-resolution input feature. The output feature value at coordinate (i, j) on the target high-resolution feature map; The filtering weights are dynamically generated for the target position (i, j) and have a relative offset of (u, v) within the corresponding local neighborhood. The input is the feature value of the neighboring pixels corresponding to the target location on the low-resolution feature map; These are the horizontal and vertical spatial offset indices within the local filter kernel; This refers to the set of local neighborhood spaces covered by the dynamic filter kernel. This process uses a dynamically generated low-pass filter to specifically attenuate high-frequency noise, mitigating the detail degradation and checkerboard effect problems caused by upsampling at their source.

[0075] 204: The LCO (Local Consistency Optimization) module performs local similarity localization and resampling correction;

[0076] like Figure 4 The LCO module flowchart shown is divided into two steps from top to bottom.

[0077] The first step is to calculate the features of the center pixel in the feature map. Its 8-neighbor pixel features Local cosine similarity The formula is as follows:

[0078]

[0079] Where, · represents the dot product of two eigenvectors, | | and | | represent the eigenvectors respectively and The L2 norm (i.e., the magnitude of the vector). The local cosine similarity between the center pixel and its k-th neighboring pixel is represented by k, where k is the index number of the 8 neighboring pixels. A local similarity matrix is ​​constructed in this way to locate the "candidate replacement pixel" with high similarity.

[0080] The second step involves using the initial fused features and the aforementioned local similarity matrix as input to predict the 2D resampling offset for each pixel through a lightweight convolutional network. Based on the predicted 2D resampling offset, deformable resampling is performed, replacing features in low-similarity regions such as object edges or noise points with features in high-similarity regions, thereby accurately aligning the feature locations to the true semantic boundaries.

[0081] 205: Multi-scale dual-pooling feature aggregation in the MSDP (Multi-scale Dual Pooling) module;

[0082] like Figure 5 The MSDP module diagram shown, viewed from left to right: High-resolution features output by the LCO module and features of different scales from skip connections are fed into dual-pooling branches. The upper branch uses max pooling to extract obvious local high-frequency textures (such as building edges); the lower branch simultaneously uses average pooling to preserve globally smooth background information. Subsequently, the pooling results are concatenated and fused with the original features, and a residual connection is used to prevent gradient vanishing during feature propagation. The core aggregation process can be represented as follows:

[0083]

[0084] in, This is the final multi-scale fusion feature representation output by the module; This represents the feature concatenation operation performed along the channel dimension; F is the input feature map to be processed (including high-resolution features output by the LCO module or skip connection features at different scales). MaxPool represents the max pooling operation, and AvgPool represents the average pooling operation. This indicates an element-wise addition operation, used to implement residual connections to prevent gradient vanishing.

[0085] This mechanism greatly enhances the network's ability to simultaneously process large background areas (such as bodies of water) and small targets (such as vehicles).

[0086] 206: Optimize training for combined loss of easily confused categories;

[0087] At the end of model training, this embodiment of the invention employs a combined loss function. This is used to guide network weight updates. The combined loss integrates the weighted cross-entropy loss responsible for pixel-by-pixel classification accuracy. And Dice loss, which is dedicated to optimizing object edges and mitigating sample imbalance. The formula is as follows:

[0088]

[0089] in, and This is the balance coefficient.

[0090] In particular, after counting the frequency of occurrence of each land cover category, the network assigned higher category weights to easily confused categories such as low-lying plants or those with a small proportion. This guides the model to increase the penalty for minority classes during training, thereby reducing feature misidentification.

[0091] Finally, through continuous iteration of forward and backward propagation, a high-precision multimodal semantic segmentation prediction map is output.

[0092]

[0093]

[0094] in, This is the total combined loss function; The weighted cross-entropy loss; For Dice's loss; and C is the balance coefficient; C is the total number of land cover categories; This represents the penalty weight assigned to the c-th type of land cover; The true label value indicating whether a pixel belongs to the c-th type of land cover (usually using one-hot encoding; 1 if it belongs to the class, 0 otherwise); This represents the predicted probability value of the pixel belonging to the c-th type of land cover, as output by the network model. N represents the total number of pixels in the image; This represents the true label value of the i-th pixel belonging to category c; This represents the predicted probability value of the i-th pixel belonging to category c. express The squared term, express The squared term.

[0095] In summary, the embodiments of the present invention effectively solve the problems of insufficient utilization of multimodal data and noise interference caused by degradation of upsampling details, and provide reliable technical support for modern urban remote sensing monitoring and refined land cover analysis.

[0096] Example 3

[0097] This embodiment provides a remote sensing image semantic segmentation device based on a multi-level multimodal architecture. The device is logically divided into multiple cooperative functional modules to achieve efficient processing and accurate classification of high-resolution remote sensing data.

[0098] First, multimodal data differential preprocessing module

[0099] This module is responsible for acquiring optical color images (RGB) and digital surface model (DSM) data of the region to be processed, and performing high-precision pixel value normalization to map the data to the [0, 1] distribution range to eliminate the influence of dimensions and accelerate model convergence. At the same time, this module integrates differentiated cutting logic for datasets of different sizes. For normalized datasets such as ISPRSVaihingen, the original shape is preserved, while for ultra-large images such as ISPRS Potsdam, the sliding window algorithm is used to cut them into 1024 × 1024 pixel image blocks, supplemented by spatial geometric enhancement methods such as random rotation and mirror flipping, so as to provide high-quality and robust training sample input for the subsequent encoder.

[0100] Second, dual-stream coding and cross-modal joint context interaction module

[0101] This module constructs a symmetrical U-shaped dual-stream path through integrated convolutional layers (based on ImageNet pre-trained ResNet50) and a Transformer module, which are used to capture local texture features of remote sensing images and model long-range global dependencies, respectively. The core of this module lies in the built-in Joint Context-Aware (JC-SE) unit, which aggregates the joint spatial information of dual-stream features through global average pooling and generates cross-modal collaborative vectors. Then, it uses fully connected activation paths to generate collaborative weights to perform dynamic channel-level recalibration of features, ensuring that each feature can perceive the global state of the other modality, thus solving the problem of color or highly similar ground objects at the architectural level.

[0102] Third, spatial adaptive decoding and noise suppression module

[0103] This module replaces the traditional interpolation mechanism with a built-in Spatial Adaptive Low-Pass Upsampling (SALU) component. It uses a lightweight convolutional network to autonomously predict the spatial variation filter weights corresponding to the local neighborhood during the decoding stage. By performing sub-pixel convolution reconstruction on the normalized filter weights, this module can dynamically attenuate high-frequency aliasing noise during the recovery of deep features to high resolution. It effectively preserves clear semantic edges while smoothing the internal features of the target, thereby solving the problems of detail degradation and feature aliasing that are easily caused by traditional decoders from the source.

[0104] Fourth, the module for local consistency optimization and precise boundary alignment.

[0105] This module integrates a Local Consistency Optimizer (LCO), which constructs a similarity matrix by calculating the feature cosine similarity between each pixel and its 8 neighboring pixels, accurately locating "candidate replacement pixels" with high similarity. Based on this, the module uses lightweight convolutional units to predict the two-dimensional resampling offset of each pixel and performs deformable resampling, replacing features in low-similarity regions such as object boundaries or noise points with features in high-similarity regions, forcing feature positions to be precisely aligned with the true semantic boundaries, effectively correcting geometric boundary deviations introduced by receptive field limitations or upsampling.

[0106] Fifth, multi-scale dual-pooling aggregation and fusion output module

[0107] This module performs parallel max pooling and average pooling operations on the optimized feature map and skip connection features at different scales through a multi-scale dual pooling (MSDP) component, aiming to simultaneously extract significant local high-frequency textures and preserve globally smooth background environmental information. This module utilizes residual connections and cross-scale fusion mechanisms to aggregate dual pooling features at different levels by channel dimension, significantly improving the device's comprehensive discrimination capability for large homogeneous regions and small complex targets. Finally, based on this fused representation, a high-precision, pixel-level semantic classification prediction map with sharp boundaries is generated.

[0108] Sixth, targeted optimization of training and hardware execution units.

[0109] The device incorporates a combined loss calculation unit consisting of weighted cross-entropy loss and Dice loss. It applies higher penalty weights to easily confused categories such as low-lying vegetation through a class balancing strategy, ensuring the model's classification balance in complex urban scenarios. At the hardware level, the device relies on a computing server equipped with an Intel Xeon Gold processor (such as the 5218R), at least 256GB of memory, and a high-performance graphics card (such as the NVIDIA A40). It runs in a Linux environment and under the Python / PyTorch deep learning framework, supporting iterative updates of network parameters and model deployment through the AdamW optimizer and cosine annealing strategy.

[0110] Example 4

[0111] A remote sensing image semantic segmentation device based on a multi-level, multimodal architecture, the device comprising:

[0112] The optical color image and digital surface model data of the region to be processed are acquired, and differential preprocessing and spatial enhancement are performed on the optical color image and digital surface model data to obtain preprocessed dual-modal image data.

[0113] Based on the preprocessed dual-modal image data, dual-stream feature extraction and cross-modal joint context recalibration are performed to obtain cross-modal recalibration features;

[0114] Based on the cross-modal recalibration features, spatial adaptive low-pass upsampling and noise suppression are performed to obtain high-quality upsampled feature maps. Local consistency optimization and boundary semantic alignment are then performed to obtain boundary-optimized feature maps.

[0115] By acquiring skip connection features at different scales, and based on the feature maps optimized by the boundary and the skip connection features at different scales, multi-scale dual pooling aggregation and multi-level feature fusion are performed to obtain pixel-level semantic prediction results.

[0116] The true pixel label values ​​of the corresponding regions of ground features are obtained. Based on the pixel-level semantic prediction results and the true pixel label values, the network parameters are optimized and trained by combining loss functions to obtain the final high-precision multimodal semantic classification map.

[0117] Specifically, based on the preprocessed dual-modal image data, dual-stream feature extraction and cross-modal joint context recalibration are performed to obtain the cross-modal recalibration features as follows:

[0118] In the feature encoding stage, a joint context-aware shallow feature fusion module is introduced. Through global average pooling, the joint global context information of the two-stream features is extracted, and cross-modal collaborative vectors are constructed accordingly. Independent fully connected activation paths are used to generate collaborative weights for each branch. Channel-level multiplication recalibration is performed on the original features using the joint weights.

[0119] Based on the cross-modal recalibration characteristics, spatial adaptive low-pass upsampling and noise suppression are performed as follows:

[0120] By utilizing a spatially adaptive low-pass upsampling module, a lightweight convolutional network autonomously predicts low-pass filter weights corresponding to spatial variations in the local neighborhood. Sub-pixel convolutional reconstruction technology is applied to the normalized filter weights to dynamically attenuate high-frequency aliasing noise and feature aliasing interference during the upsampling of calibrated multimodal features.

[0121] Among them, local consistency optimization and boundary semantic alignment are:

[0122] The upsampled feature map is input into local consistency optimization, and the feature cosine similarity between each pixel and its 8 neighboring pixels is calculated to construct a local similarity matrix. Based on the initial fused features and similarity matrix, a lightweight convolutional network is used to predict the resampling offset direction and magnitude of each pixel in two-dimensional space, and deformable resampling operation is performed.

[0123] The cross-modal cooperation vector is:

[0124] Extracted RGB features and DSM characteristics Global average pooling is used to aggregate the global spatial information of the two streams. The aggregated features are then concatenated to generate a joint context vector for cross-modal cooperation. The calculation formula is as follows:

[0125] .

[0126] The calibrated multimodal features are as follows:

[0127]

[0128] Where X represents the low-resolution input feature. The output feature value at coordinate (i, j) on the target high-resolution feature map; The filtering weights are dynamically generated for the target position (i, j) and have a relative offset of (u, v) within the corresponding local neighborhood. The input is the feature value of the neighboring pixels corresponding to the target location on the low-resolution feature map; These are the horizontal and vertical spatial offset indices within the local filter kernel; It is the set of local neighborhood spaces covered by the dynamic filter kernel.

[0129] Among them, multi-scale dual-pooling polymerization is as follows:

[0130] The upper branch uses max pooling to extract obvious local high-frequency textures; the lower branch uses average pooling to preserve globally smooth background information.

[0131] The two pooling results are concatenated and fused with the original features, and residual connections are used. The aggregation process is represented as follows:

[0132]

[0133] in, This is for the final multi-scale fusion feature representation; This represents the feature concatenation operation performed along the channel dimension; F is the input feature map to be processed.

[0134] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0135] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0136] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0137] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0138] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0139] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0140] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0141] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0142] Example 5

[0143] To verify the effectiveness of the JAD-TransUNet framework described in this embodiment of the invention in complex remote sensing scenarios, this embodiment conducted simulation verification in accordance with standard scientific experimental procedures.

[0144] 1. Experimental Objective

[0145] The effectiveness of the JC-SE module, SALU module, LCO module, and MSDP module proposed in this invention in improving the semantic segmentation accuracy (especially boundary accuracy) of remote sensing images was verified.

[0146] 2. Experimental Environment and Dataset Setup Hardware Environment: A computing server equipped with an Intel Xeon Gold 5218R processor, 256GB of RAM, and a high-performance NVIDIA A40 graphics card. Software Environment: Linux operating system, Python programming language, and PyTorch deep learning framework. Dataset: The publicly available ISPRS Vaihingen and ISPRS Potsdam high-resolution remote sensing land cover datasets were selected.

[0147] 3. Evaluation metrics adopted are pixel-level metrics commonly used in the remote sensing field: Overall Accuracy (OA): assesses the proportion of correct classifications across the entire map. Mean Intersection over Union (mIoU): assesses the degree of overlap between the predicted and actual regions. F1 Score: comprehensively evaluates precision and recall.

[0148] 4. Experimental Results and Comparative Analysis

[0149] 4.1 Comparison with mainstream algorithms

[0150] The method of this invention was compared with current state-of-the-art convolutional neural network (CNN) models and Transformer models on the ISPRS Vaihingen dataset. The results are shown in Table 1:

[0151]

[0152] Results Analysis: Experimental data show that the method of this invention significantly outperforms the traditional model in terms of overall performance. This is attributed to the deep synergistic utilization of RGB and DSM data by the dual-stream coding architecture, as well as the effective suppression of noise during the upsampling process.

[0153] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0154] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-level, multi-modal remote sensing image semantic segmentation method, characterized in that, The method includes: The optical color image and digital surface model data of the region to be processed are acquired, and differential preprocessing and spatial enhancement are performed on the optical color image and digital surface model data to obtain preprocessed dual-modal image data. Based on the preprocessed dual-modal image data, dual-stream feature extraction and cross-modal joint context recalibration are performed to obtain cross-modal recalibration features; Based on the cross-modal recalibration features, spatial adaptive low-pass upsampling and noise suppression are performed to obtain high-quality upsampled feature maps. Local consistency optimization and boundary semantic alignment are then performed to obtain boundary-optimized feature maps. By acquiring skip connection features at different scales, and based on the feature maps optimized by the boundary and the skip connection features at different scales, multi-scale dual pooling aggregation and multi-level feature fusion are performed to obtain pixel-level semantic prediction results. The true pixel label values ​​of the corresponding regions of ground features are obtained. Based on the pixel-level semantic prediction results and the true pixel label values, the network parameters are optimized and trained by combining loss functions to obtain the final high-precision multimodal semantic classification map.

2. The remote sensing image semantic segmentation method based on multi-level multimodal as described in claim 1, characterized in that, Based on the preprocessed bimodal image data, dual-stream feature extraction and cross-modal joint context recalibration are performed to obtain the cross-modal recalibration features as follows: In the feature encoding stage, a joint context-aware shallow feature fusion module is introduced. Through global average pooling, the joint global context information of the two-stream features is extracted, and cross-modal collaborative vectors are generated accordingly. Independent fully connected activation paths are used to generate collaborative weights for each branch. Channel-level multiplication recalibration is performed on the original features using the joint weights.

3. The remote sensing image semantic segmentation method based on multi-level multimodal as described in claim 1, characterized in that, The spatial adaptive low-pass upsampling and noise suppression based on cross-modal recalibration characteristics is as follows: By utilizing a spatially adaptive low-pass upsampling module, a lightweight convolutional network autonomously predicts low-pass filter weights corresponding to spatial variations in the local neighborhood. Sub-pixel convolutional reconstruction technology is applied to the normalized filter weights to dynamically attenuate high-frequency aliasing noise and feature aliasing interference during the upsampling of calibrated multimodal features.

4. The remote sensing image semantic segmentation method based on multi-level multimodal as described in claim 1, characterized in that, The local consistency optimization and boundary semantic alignment are as follows: The upsampled feature map is input into local consistency optimization, and the feature cosine similarity between each pixel and its 8 neighboring pixels is calculated to construct a local similarity matrix. Based on the initial fused features and similarity matrix, a lightweight convolutional network is used to predict the resampling offset direction and magnitude of each pixel in two-dimensional space, and deformable resampling operation is performed.

5. The remote sensing image semantic segmentation method based on multi-level multimodal as described in claim 2, characterized in that, The cross-modal cooperation vector is: Extracted RGB features and DSM characteristics Global average pooling is used to aggregate the global spatial information of the two streams. The aggregated features are then concatenated to generate a joint context vector for cross-modal cooperation. The calculation formula is as follows: 。 6. The remote sensing image semantic segmentation method based on multi-level multimodal as described in claim 3, characterized in that, The calibrated multimodal features are as follows: ； Where X represents the low-resolution input feature. The output feature value at coordinate (i, j) on the target high-resolution feature map; The filtering weights are dynamically generated for the target position (i, j) and have a relative offset of (u, v) within the corresponding local neighborhood. The input is the feature value of the neighboring pixels corresponding to the target location on the low-resolution feature map; These are the horizontal and vertical spatial offset indices within the local filter kernel; It is the set of local neighborhood spaces covered by the dynamic filter kernel.

7. The remote sensing image semantic segmentation method based on multi-level multimodal as described in claim 4, characterized in that, The multi-scale dual-pooling polymerization is as follows: The upper branch uses max pooling to extract obvious local high-frequency textures; the lower branch uses average pooling to preserve globally smooth background information. The two pooling results are concatenated and fused with the original features, and residual connections are used. The aggregation process is represented as follows: ； in, This is for the final multi-scale fusion feature representation; This represents the feature concatenation operation performed along the channel dimension; F is the input feature map to be processed.

8. A remote sensing image semantic segmentation device based on multi-level multimodal methods, the device comprising: A processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the apparatus to perform the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-7.

Images