Road environment semantic segmentation method based on cross-modal difference modulation and visual basis model

Abstract

The invention discloses a road environment semantic segmentation method based on cross-modal difference modulation and a visual basic model, and the method is based on a DINOv3 visual basic model of a frozen weight, and introduces a complementary difference fusion module and a bidirectional context flow alignment module through the design of double-flow frozen coding and difference perception interaction. According to the method, a cross attention weight is generated by calculating a local high-frequency difference, so that transverse anti-noise complementation of RGB texture information and a Depth geometric structure is realized, and single-mode noise pollution is effectively inhibited; the two-way cross-scale alignment mechanism of top-down space mask guidance and bottom-up channel feature feedback is constructed by the two-way cross-scale alignment mechanism, and the scale barrier of deep and shallow layer features is broken. Therefore, joint feature representation with deep synergy of semantics and structures is obtained, inhibition of the model to environmental noise and reservation of tiny object details are balanced in a complex road scene, and pixel-level semantic segmentation with high generalization and high precision is realized.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving assistance, and more particularly to a road environment semantic segmentation method for cross-modal difference modulation and visual basis model for complex road environments. Background Technology

[0002] With the rapid development of intelligent transportation systems and autonomous driving technology, the ability of vehicles to perceive complex road environments around the clock and with high precision has become a core element in ensuring driving safety. In autonomous driving visual perception systems, semantic segmentation undertakes the important task of "pixel-level classification," that is, it needs to accurately identify and segment various targets such as roads, vehicles, pedestrians, and traffic signs to provide a detailed semantic map for subsequent path planning and decision control. Traditional methods mainly rely on manually designed feature extraction operators, but their robustness and accuracy often fail to meet the needs of practical applications when faced with drastic changes in lighting, inclement weather, and complex unstructured road scenarios.

[0003] The rise of deep learning technology, especially the widespread application of convolutional neural networks (CNNs) and the Transformer architecture, has greatly improved the accuracy of semantic segmentation. However, relying solely on single RGB modal data is prone to false positives or false negatives when visual information is impaired (e.g., in low light at night, glare from strong light, or when colors and textures are similar). Therefore, introducing depth map data containing rich 3D geometric information to construct an RGB-D multimodal fusion perception system has become an inevitable trend for improving the environmental adaptability of autonomous driving. RGB modality provides appearance texture, while depth modality provides spatial structure; their complementarity offers a new opportunity to solve the segmentation challenges in complex scenes.

[0004] As autonomous driving technology extends from closed parks to open urban roads and even unstructured rural roads, the environmental complexity faced by perception systems is increasing exponentially. Traditional semantic segmentation methods heavily rely on a single RGB camera, making them prone to missed or false detections in nighttime, glare-filled, or shadowy scenes. Introducing depth maps to construct RGB-D multimodal perception systems has become an industry consensus. However, existing RGB-D fusion schemes mainly employ a "dual-stream encoding-post-fusion" or "feature pyramid fusion" architecture. Investigations have revealed the following core flaws in existing technologies:

[0005] (1) Additive noise propagation: Existing fusion methods (such as direct stitching or simple cross attention) lack the ability to judge the quality of features within a modality. When sensor noise or artifacts exist in a certain modality (such as depth map), these interference information will be fused into the feature stream indiscriminately, destroying the effective features of another modality (such as RGB map), resulting in jagged or sticky edges in the final segmentation result.

[0006] (2) Unidirectionality of scale interaction: Classic U-Net or FPN architectures typically only contain top-down feature propagation paths. While deep semantic features are accurate in class identification, they have low spatial resolution; while shallow features have high resolution, they lack semantic guidance and are prone to misclassifying water stains on the road surface as obstacles. The lack of a bottom-up feedback mechanism makes it impossible for deep features to perceive subtle texture changes.

[0007] (3) Pre-training on specific datasets limits the generalization of the model: Existing solutions generally use backbone networks that are pre-trained in a supervised manner on classification datasets such as ImageNet. Such networks focus more on specific category textures and lack an understanding of general visual structures, resulting in a significant performance drop and insufficient robustness when the model is transferred from urban roads to rural roads or from sunny days to rainy or foggy days. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a road environment semantic segmentation method based on cross-modal differential modulation and visual basic model, which aims to improve the noise resistance, boundary accuracy and scene generalization ability of autonomous driving perception system in complex environments.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental models mainly includes the following specific steps:

[0011] Step S1: Complete the semantic segmentation model construction and train it using a public dataset;

[0012] Step S2: Obtain the RGB image and corresponding depth image of the driving scene as input to the model;

[0013] Step S3: Input the RGB and HHA-encoded depth maps into the frozen DINOv3 backbone network and extract multi-scale general semantic features respectively;

[0014] Step S4: Enter the complementary difference fusion module: calculate high-frequency difference features and generate texture and geometric attention maps;

[0015] Step S5: Perform bidirectional cross-injection: geometrically enhance RGB texture, refine texture depth geometry, and output blended features;

[0016] Step S6: Enter the bidirectional context flow alignment module: perform closed-loop mutual refinement using deep spatial mask and shallow channel feedback;

[0017] Step S7: Input the optimized multi-scale features into the decoder for step-by-step upsampling and feature recovery;

[0018] Step S8: Output pixel-level semantic classification results to generate the final road environment semantic segmentation results.

[0019] Furthermore, step S3 includes the following steps:

[0020] Step S31: Using the HHA encoding strategy, the depth map is converted into a three-channel geometric image; the three channels are: channel 1, representing horizontal parallax, used to simulate parallax information in binocular vision, which is extremely sensitive to object edges; channel 2, representing ground height, used to represent height information after calibration using the gravity direction; and channel 3, representing the angle of the normal vector, used to represent the angle between the surface normal vector and the gravity direction. The geometric image after HHA encoding is aligned with the RGB image in terms of data dimension, and in terms of information content, the implicit distance information is made explicit as geometric structure information.

[0021] Furthermore, step S3 also includes the following steps:

[0022] Step S32: Input the RGB image and the HHA encoded geometric image into two DINOv3 networks with the same structure but independent from each other. During the entire training process, all parameters of DINOv3 are kept frozen, that is, the gradient is not backpropagated. Then, the output of the 3rd, 6th, 9th and 12th layers of DINOv3 are extracted as the input of the feature pyramid. Then, the resolution is adjusted to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image by bilinear interpolation.

[0023] Furthermore, step S4 includes the following steps:

[0024] Step S41: Perform modality-specific preprocessing on the reconstructed features from step S3: RGB branch uses... Convolution performs channel mapping, preserving global semantics, denoted as The HHA branch adopts Depthwise separable convolution is used to enhance geometric edges, denoted as... ;

[0025] Step S42: At each level of the encoder, to address the issue of noise contamination between modes, a complementary difference fusion module is introduced for lateral feature fusion. This module first calculates high-frequency difference features, defining kernel sizes as follows: and Average pooling operation; extracting high-frequency details by subtracting aggregated context features from the original features. Taking the RGB branch as an example, differential features... The calculation is shown in the following formula:

[0026]

[0027] It can respond to extremely subtle texture changes, but also includes most of the sensor speckle noise; The response to structural edges at larger scales has a certain smoothing and suppression effect on isolated noise points. This method concatenates the two in the channel dimension to construct a multi-scale frequency domain descriptor. The subsequent convolutional network will use this descriptor to determine whether a high-frequency response belongs to "structural edge" or "random noise".

[0028] Step S43: Generate a texture attention map using convolution and the Sigmoid function. and geometric attention graph .

[0029] Furthermore, step S5 includes the following steps:

[0030] Perform bidirectional cross-injection: leveraging geometric attention Injecting HHA geometric features into the RGB branch to fix shadow blur; utilizing texture attention Injecting RGB features into the HHA branch to suppress edge noise; finally fusing features The calculation is shown in the following formula:

[0031]

[0032] In the formula, the convolutional layer in fused is It is a learnable convolutional layer. , The scaling factor is learnable; RGB features are injected into the depth branch. Since the depth map itself is smooth in the shadows, the texture features provided by RGB can help the depth branch refine the road surface segmentation boundary and prevent the segmentation from becoming blurry due to the lack of texture in the depth branch.

[0033] Furthermore, step S6 includes the following steps:

[0034] The multi-scale fused features, processed by the bidirectional context flow alignment module, are input into the decoder module. The decoder adopts a bottom-up, step-by-step reconstruction path, starting from the deepest layer and gradually restoring the spatial resolution of the feature map through a series of upsampling units and the bidirectional context flow alignment module. The specific interaction mechanism is as follows:

[0035] Step S61: Generate a space mask from top to bottom:

[0036] First, deep features shallow features The splicing is performed along the channel dimension, first passing through a... The convolutional layer performs channel dimensionality reduction and preliminary feature fusion, followed by dilated convolution to expand the receptive field, and then generates a spatial mask after sigmoid activation. This mask uses deep semantic information to indicate the spatial location of the foreground object, as shown in the following formula:

[0037]

[0038] Step S62: Bottom-up feedback of channel weights:

[0039] Using the generated shallow features Masking is performed to retain only the features of the foreground region. Then, global average pooling and multilayer perceptron processing are used to generate channel attention vectors. This vector reflects the importance of each feature channel within the key region, as shown in the following equation:

[0040]

[0041] Shallow features often contain a lot of irrelevant background textures, which this method addresses by multiplying by... It also uses deep semantics to forcibly suppress the response of non-target regions in shallow features, which is equivalent to performing a semantic-based spatial denoising.

[0042] Step S63: Bidirectional Feature Calibration:

[0043] The calculated weights are applied back to their respective input paths to achieve deep semantic channel activation and shallow spatial noise filtering, ultimately outputting the calibrated features. The calculation process is shown in the following formula:

[0044]

[0045] In the formula It is a reverse guidance; this allows deep semantics to maintain its abstractness while being sensitive to the texture of specific instances.

[0046] Furthermore, step S7 includes the following steps:

[0047] After the decoder outputs the last layer of features, a prediction and reconstruction step is performed, including segmentation projection and final upsampling, as follows:

[0048] Step S71: Segment the projection, using a convolution kernel of size... The convolutional layer is used as the prediction head, mapping the number of feature channels in the final output of the bidirectional context flow alignment module to the number of semantic categories. This yields a preliminary semantic probability map;

[0049] Step S72: Final upsampling, since the output resolution of the prediction head is still the same as the original image. To obtain pixel-level prediction results consistent with the input image, a bilinear interpolation algorithm is used to enlarge the semantic probability map by 4 times, restoring it to the original image size. This leads to the final semantic segmentation result of the road environment.

[0050] Furthermore, the semantic segmentation method also includes the construction of a loss function:

[0051] To optimize model parameters, ensure effective learning of intermediate layer features, and enhance the model's ability to perceive object boundaries, this method employs a multi-task deep supervision strategy: the total loss function is composed of the main segmentation loss. Auxiliary segmentation loss and boundary consistency loss The calculation formula is as follows: The three-part weighted construction is shown in the following equation:

[0052]

[0053] In the formula and This is the balance coefficient.

[0054] Furthermore, the construction of the loss function also includes the following steps:

[0055] The main segmentation loss The standard cross-entropy loss is calculated at the prediction plot of the final model output, as shown in the following formula:

[0056]

[0057] The auxiliary segmentation loss To add an auxiliary classification head to deeper layers of the network, forcing the deep network to possess semantic discriminative ability early in feature propagation, effectively alleviating the gradient vanishing problem and accelerating model convergence, an auxiliary classification head is added at the output of the penultimate bidirectional context flow alignment module of the decoder, i.e., at an image resolution of [missing information]. At this point, cross-entropy loss is used for calculation.

[0058] Furthermore, the construction of the loss function also includes the following steps:

[0059] Boundary consistency loss Introducing weighted binary cross-entropy combined with IoU: First, the Laplacian operator is used to analyze the predicted semantic graph. and real labels Extracting edge probability maps and Binary cross-entropy loss:

[0060] (8)

[0061] In the formula The positive and negative sample balance coefficient is used, where edge pixels are much fewer than non-edge pixels. ;

[0062] Then the cross-union ratio (IoU) loss between the two is calculated, as shown in the following formula:

[0063]

[0064] Total marginal loss This loss function forces the object contours output by the model to be highly consistent with the real contours, significantly improving the sharpness of the segmentation edges.

[0065] The working process and principle of this invention are as follows: Based on the DINOv3 visual model with frozen weights, this invention introduces a complementary difference fusion module (CDFM) and a bidirectional context flow alignment module (BCFA) through a core design of "dual-stream frozen encoding + difference-aware interaction". The former generates cross-attention weights by calculating local high-frequency differences within a modality, achieving lateral noise resistance complementarity between RGB texture information and depth geometry, effectively suppressing single-modal noise pollution; the latter constructs a bidirectional cross-scale alignment mechanism of "top-down spatial mask guidance" and "bottom-up channel feature feedback" in the decoding and reconstruction stage, breaking down the scale barrier between deep and shallow features. This results in a joint feature representation that coordinates semantic and structural depth, balancing the model's suppression of environmental noise and preservation of minute object details in complex lighting and unstructured road scenes, achieving high generalization and high-precision pixel-level semantic segmentation.

[0066] Compared with the prior art, the present invention also has the following advantages:

[0067] (1) The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model provided by this invention addresses the noise contamination problem by utilizing a Complementary Differential Fusion Module (CDFM) to explicitly extract complementary details of texture and geometry by calculating local high-frequency differences within each modality. Employing the intermodulation strategy of "geometry-enhanced texture" and "texture-refined geometry," it achieves noise filtering and feature enhancement based on physical characteristics, ensuring that only effective information is fused.

[0068] (2) The road environment semantic segmentation method of cross-modal differential modulation and visual basic model provided by this invention addresses the problem of insufficient detail segmentation accuracy by introducing a bi-directional context-flow alignment module (BCFA) to construct a bi-directional cross-scale alignment mechanism of "top-down spatial localization" and "bottom-up channel feedback". By guiding shallow denoising with deep semantics and using shallow details to activate deep feature channels, the model's ability to perceive small objects and complex boundaries is significantly improved.

[0069] (3) The road environment semantic segmentation method based on cross-modal differential modulation and visual base model provided in this invention addresses the problem of poor model generalization by using the DINOv3 visual base model with frozen weights as a feature extractor. By utilizing the general visual representation learned on large-scale unlabeled data, it replaces the traditional supervised backbone network, significantly enhancing the robustness and adaptability of the model in unknown scenes and diverse lighting conditions.

[0070] (4) The road environment semantic segmentation method based on cross-modal differential modulation and visual base model provided by this invention addresses the problem of high training costs for large-scale models. By freezing the parameters of the DINOv3 backbone network and only fine-tuning the decoder and fusion module, it significantly reduces the GPU memory usage and computational resource consumption during model training. This "efficient parameter fine-tuning" strategy not only accelerates the convergence speed of the model but also avoids the catastrophic forgetting problem that may result from fine-tuning on small-scale driving scenario data, achieving the best balance between model performance and training cost. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the operation flow of the road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model provided by the present invention.

[0072] Figure 2 This is a schematic diagram of the model architecture provided by the present invention.

[0073] Figure 3 This is a schematic diagram of the Complementary Differential Fusion Module (CDFM) provided by the present invention.

[0074] Figure 4 This is a schematic diagram of the structure of the Bidirectional Context Flow Alignment Module (BCFA) provided by the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the present invention clearer and more explicit, the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0076] The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental models provided in this invention is widely applicable to complex road environment semantic segmentation tasks in fields such as autonomous driving perception, intelligent traffic monitoring, and advanced driver assistance systems (ADAS). It is particularly suitable for processing large-scale real-world urban scene data with the following characteristics:

[0077] 1. Rich Environmental Diversity: Applicable to outdoor road scenes covering different geographical regions and various seasonal climates. This method can effectively cope with the differences in scene appearance caused by seasonal changes, light variations, and vegetation growth, and has strong environmental robustness.

[0078] 2. High-resolution image processing: Suitable for processing high-resolution street view images. Taking advantage of the rich details contained in high-resolution inputs, this invention utilizes a cross-modal feature interaction mechanism to accurately capture small distant targets (such as traffic poles and distant pedestrians) and subtle geometric boundaries.

[0079] Dense multi-class segmentation: Suitable for tasks requiring fine-grained scene analysis, such as densely pixel-level labeled scenes containing nearly 20 or more categories (covering road surfaces, traffic facilities, natural landscapes, traffic participants, etc.). Addressing the common category confusion problem in road environments, this method can effectively distinguish regions with similar visual features but different semantics (such as sidewalks and carriageways) using multimodal information.

[0080] Example 1:

[0081] like Figures 1 to 4 As shown in the figure, this embodiment discloses a road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental models, which mainly includes the following specific steps:

[0082] Step S1: Complete the semantic segmentation model construction and train it using a public dataset;

[0083] Step S2: Obtain the RGB image and corresponding depth image of the driving scene as input to the model;

[0084] Step S3: Input the RGB and HHA-encoded depth maps into the frozen DINOv3 backbone network and extract multi-scale general semantic features respectively;

[0085] Step S4: Enter the complementary difference fusion module: calculate high-frequency difference features and generate texture and geometric attention maps;

[0086] Step S5: Perform bidirectional cross-injection: geometrically enhance RGB texture, refine texture depth geometry, and output blended features;

[0087] Step S6: Enter the bidirectional context flow alignment module: perform closed-loop mutual refinement using deep spatial mask and shallow channel feedback;

[0088] Step S7: Input the optimized multi-scale features into the decoder for step-by-step upsampling and feature recovery;

[0089] Step S8: Output pixel-level semantic classification results to generate the final road environment semantic segmentation results.

[0090] Furthermore, step S3 includes the following steps:

[0091] Step S31: Using the HHA encoding strategy, the depth map is converted into a three-channel geometric image; the three channels are: channel 1, representing horizontal parallax, used to simulate parallax information in binocular vision, which is extremely sensitive to object edges; channel 2, representing ground height, used to represent height information after calibration using the gravity direction; and channel 3, representing the angle of the normal vector, used to represent the angle between the surface normal vector and the gravity direction. The geometric image after HHA encoding is aligned with the RGB image in terms of data dimension, and in terms of information content, the implicit distance information is made explicit as geometric structure information.

[0092] Furthermore, step S3 also includes the following steps:

[0093] Step S32: Input the RGB image and the HHA encoded geometric image into two DINOv3 networks with the same structure but independent from each other. During the entire training process, all parameters of DINOv3 are kept frozen, that is, the gradient is not backpropagated. Then, the output of the 3rd, 6th, 9th and 12th layers of DINOv3 are extracted as the input of the feature pyramid. Then, the resolution is adjusted to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image by bilinear interpolation.

[0094] Furthermore, step S4 includes the following steps:

[0095] Step S41: Perform modality-specific preprocessing on the reconstructed features from step S3: RGB branch uses... Convolution performs channel mapping, preserving global semantics, denoted as The HHA branch adopts Depthwise separable convolution is used to enhance geometric edges, denoted as... ;

[0096] Step S42: At each level of the encoder, to address the issue of noise contamination between modes, a complementary difference fusion module is introduced for lateral feature fusion. This module first calculates high-frequency difference features, defining kernel sizes as follows: and Average pooling operation; extracting high-frequency details by subtracting aggregated context features from the original features. Taking the RGB branch as an example, differential features... The calculation is shown in the following formula:

[0097]

[0098] It can respond to extremely subtle texture changes, but also includes most of the sensor speckle noise; The response to structural edges at larger scales has a certain smoothing and suppression effect on isolated noise points. This method concatenates the two in the channel dimension to construct a multi-scale frequency domain descriptor. The subsequent convolutional network will use this descriptor to determine whether a high-frequency response belongs to "structural edge" or "random noise".

[0099] Step S43: Generate a texture attention map using convolution and the Sigmoid function. and geometric attention graph .

[0100] Furthermore, step S5 includes the following steps:

[0101] Perform bidirectional cross-injection: leveraging geometric attention Injecting HHA geometric features into the RGB branch to fix shadow blur; utilizing texture attention Injecting RGB features into the HHA branch to suppress edge noise; finally fusing features The calculation is shown in the following formula:

[0102]

[0103] In the formula, the convolutional layer in fused is It is a learnable convolutional layer. , The scaling factor is learnable; RGB features are injected into the depth branch. Since the depth map itself is smooth in the shadows, the texture features provided by RGB can help the depth branch refine the road surface segmentation boundary and prevent the segmentation from becoming blurry due to the lack of texture in the depth branch.

[0104] Furthermore, step S6 includes the following steps:

[0105] The multi-scale fused features, processed by the bidirectional context flow alignment module, are input into the decoder module. The decoder adopts a bottom-up, step-by-step reconstruction path, starting from the deepest layer and gradually restoring the spatial resolution of the feature map through a series of upsampling units and the bidirectional context flow alignment module. The specific interaction mechanism is as follows:

[0106] Step S61: Generate a space mask from top to bottom:

[0107] First, deep features shallow features The splicing is performed along the channel dimension, first passing through a... The convolutional layer performs channel dimensionality reduction and preliminary feature fusion, followed by dilated convolution to expand the receptive field, and then generates a spatial mask after sigmoid activation. This mask uses deep semantic information to indicate the spatial location of the foreground object, as shown in the following formula:

[0108]

[0109] Step S62: Bottom-up feedback of channel weights:

[0110] Using the generated shallow features Masking is performed to retain only the features of the foreground region. Then, global average pooling and multilayer perceptron processing are used to generate channel attention vectors. This vector reflects the importance of each feature channel within the key region, as shown in the following equation:

[0111]

[0112] Shallow features often contain a lot of irrelevant background textures, which this method addresses by multiplying by... It also uses deep semantics to forcibly suppress the response of non-target regions in shallow features, which is equivalent to performing a semantic-based spatial denoising.

[0113] Step S63: Bidirectional Feature Calibration:

[0114] The calculated weights are applied back to their respective input paths to achieve deep semantic channel activation and shallow spatial noise filtering, ultimately outputting the calibrated features. The calculation process is shown in the following formula:

[0115]

[0116] In the formula It is a reverse guidance; this allows deep semantics to maintain its abstractness while being sensitive to the texture of specific instances.

[0117] Furthermore, step S7 includes the following steps:

[0118] After the decoder outputs the last layer of features, a prediction and reconstruction step is performed, including segmentation projection and final upsampling, as follows:

[0119] Step S71: Segment the projection, using a convolution kernel of size... The convolutional layer is used as the prediction head, mapping the number of feature channels in the final output of the bidirectional context flow alignment module to the number of semantic categories. This yields a preliminary semantic probability map;

[0120] Step S72: Final upsampling, since the output resolution of the prediction head is still the same as the original image. To obtain pixel-level prediction results consistent with the input image, a bilinear interpolation algorithm is used to enlarge the semantic probability map by 4 times, restoring it to the original image size. This leads to the final semantic segmentation result of the road environment.

[0121] Furthermore, the semantic segmentation method also includes the construction of a loss function:

[0122] To optimize model parameters, ensure effective learning of intermediate layer features, and enhance the model's ability to perceive object boundaries, this method employs a multi-task deep supervision strategy: the total loss function is composed of the main segmentation loss. Auxiliary segmentation loss and boundary consistency loss The calculation formula is as follows: The three-part weighted construction is shown in the following equation:

[0123]

[0124] In the formula and This is the balance coefficient.

[0125] Furthermore, the construction of the loss function also includes the following steps:

[0126] The main segmentation loss The standard cross-entropy loss is calculated at the prediction plot of the final model output, as shown in the following formula:

[0127]

[0128] The auxiliary segmentation loss To add an auxiliary classification head to deeper layers of the network, forcing the deep network to possess semantic discriminative ability early in feature propagation, effectively alleviating the gradient vanishing problem and accelerating model convergence, an auxiliary classification head is added at the output of the penultimate bidirectional context flow alignment module of the decoder, i.e., at an image resolution of [missing information]. At this point, cross-entropy loss is used for calculation.

[0129] Furthermore, the construction of the loss function also includes the following steps:

[0130] Boundary consistency loss Introducing weighted binary cross-entropy combined with IoU: First, the Laplacian operator is used to analyze the predicted semantic graph. and real labels Extracting edge probability maps and Binary cross-entropy loss:

[0131] (8)

[0132] In the formula The positive and negative sample balance coefficient is used, where edge pixels are much fewer than non-edge pixels. ;

[0133] Then the cross-union ratio (IoU) loss between the two is calculated, as shown in the following formula:

[0134]

[0135] Total marginal loss This loss function forces the object contours output by the model to be highly consistent with the real contours, significantly improving the sharpness of the segmentation edges.

[0136] The working process and principle of this invention are as follows: Based on the DINOv3 visual model with frozen weights, this invention introduces a complementary difference fusion module (CDFM) and a bidirectional context flow alignment module (BCFA) through a core design of "dual-stream frozen encoding + difference-aware interaction". The former generates cross-attention weights by calculating local high-frequency differences within a modality, achieving lateral noise resistance complementarity between RGB texture information and depth geometry, effectively suppressing single-modal noise pollution; the latter constructs a bidirectional cross-scale alignment mechanism of "top-down spatial mask guidance" and "bottom-up channel feature feedback" in the decoding and reconstruction stage, breaking down the scale barrier between deep and shallow features. This results in a joint feature representation that coordinates semantic and structural depth, balancing the model's suppression of environmental noise and preservation of minute object details in complex lighting and unstructured road scenes, achieving high generalization and high-precision pixel-level semantic segmentation.

[0137] Example 2:

[0138] like Figures 1 to 4 As shown, this embodiment proposes an RGB-D semantic segmentation model for road environments based on cross-modal differential modulation and a visual foundation model. This model uses the DINOv3 visual foundation model with frozen weights as its core, extracting features from RGB color images and HHA-encoded depth images. By introducing a complementary differential fusion module (CDFM), lateral noise resistance and complementarity between modalities are achieved during the encoding stage. Simultaneously, a bidirectional context flow alignment module (BCFA) is introduced during the decoding stage to construct a top-down spatial guidance and a bottom-up channel feedback mechanism, breaking down the scale barrier between shallow and deep features, thereby improving the segmentation accuracy of autonomous driving systems for small objects and boundaries in complex unstructured scenes. The following is a detailed description of the technical solution of this embodiment:

[0139] This implementation constructs a deep learning model architecture capable of processing high-resolution, multimodal data. For example... Figure 2 As shown, the model mainly consists of three parts: input preprocessing and dual-stream backbone network, complementary differential fusion encoder, and bidirectional aligned decoder.

[0140] (1) The physical meaning and necessity of HHA coding

[0141] The raw depth map received by the system is typically a single-channel 16-bit integer or 32-bit floating-point matrix, with values ​​representing the physical distance to the distance sensor. However, the backbone network DINOv3 used in this embodiment is a Transformer architecture pre-trained on RGB images (three channels). Directly inputting a single-channel depth map into DINOv3 leads to a dimensionality mismatch in the input layer and fails to fully utilize DINOv3's feature extraction capabilities for color textures. Therefore, this embodiment employs an HHA encoding strategy to convert the depth map into a three-channel geometric image:

[0142] Channel 1 (Horizontal Parallax): Simulates parallax information in binocular vision and is extremely sensitive to object edges.

[0143] Channel 2 (Ground Elevation): Utilizes gravity-calibrated height information. This is crucial for road segmentation, as road surfaces typically have a consistently low height value, while vehicles and pedestrians have higher height values. This provides the model with a strong semantic prior independent of lighting conditions.

[0144] Channel 3 (Normal Angle): The angle between the surface normal vector and the direction of gravity. Road surfaces are typically close to 0 degrees or 180 degrees, while vertical walls or vehicle sides are close to 90 degrees. Through HHA encoding, the geometric image is aligned with the RGB image in terms of data dimension, and the implicit distance information is made explicit as geometric structure information in terms of information content, enabling the frozen DINOv3 backbone network to process the "geometric image" like a "color image".

[0145] (2) DINOv3 backbone network with frozen weights

[0146] This embodiment uses the DINOv3 model as the feature extractor. Unlike traditional supervised training models, DINOv3 is trained on a large-scale dataset through self-supervised learning. Its core advantage lies in its ability to capture general object shape and texture features, rather than being limited to specific semantic categories. In implementation, we input RGB images and HHA-encoded geometric images into two structurally identical but independent DINOv3 networks.

[0147] Throughout the training process, all parameters of DINOv3 are frozen, meaning gradients are not backpropagated. This serves two purposes: First, it ensures generalization, preventing fine-tuning on small-scale RGB-D datasets from causing the model to forget general features learned on large-scale data, thus guaranteeing the model's robustness in unfamiliar road environments. Second, it significantly reduces computational efficiency, drastically reducing memory usage and computational cost during backpropagation. We extract the outputs of layers 3, 6, 9, and 12 of DINOv3 as input to the feature pyramid, and then use bilinear interpolation to adjust the resolution to match the original image. .

[0148] (3) Complementary Difference Fusion Module (CDFM) and Horizontal Interaction:

[0149] Modality-specific preprocessing is performed on the reconstructed features in (1): the RGB branch adopts... Convolution performs channel mapping, preserving global semantics, denoted as The HHA branch adopts Depthwise separable convolution (DWConv) is used to enhance geometric edges, denoted as... .

[0150] To address the issue of noise contamination between modes at each level of the encoder, a complementary difference fusion module (CDFM) is introduced for lateral feature fusion. This module first calculates high-frequency difference features, defining kernel sizes as follows: and The average pooling operation (AvgPool) extracts high-frequency details by subtracting the aggregated context features from the original features. Taking the RGB branch as an example, the difference features... The calculation is shown in equation (1):

[0151]

[0152] It can respond to extremely subtle texture changes, but also includes most of the sensor speckle noise. The response to structural edges at larger scales has a certain smoothing and suppression effect on isolated noise points. This embodiment creatively splices the two at the channel dimension to construct a multi-scale frequency domain descriptor. Subsequent convolutional networks will use this descriptor to determine whether a high-frequency response belongs to "structural edge" (response at both scales) or "random noise" (response only at a small scale).

[0153] Subsequently, a texture attention map is generated using convolution and the Sigmoid function. and geometric attention graph Based on this, bidirectional cross-injection is performed: utilizing geometric attention. Instruct HHA geometric feature injection into RGB branches to fix shadow blur, utilizing texture attention. Injecting RGB features into the HHA branch to suppress edge noise. Final feature fusion. The calculation is shown in equation (2):

[0154]

[0155] The convolutional layer in fused is It is a learnable convolutional layer. , The scaling factor is learnable because it first considers the presence of jagged edge noise in the depth map. In jagged regions, the high-frequency differences in the depth map are large, which would normally be considered edges. However, in CDFM... Through training, it learns this jagged, high-frequency distribution characteristic (usually discontinuous, with frequent high-frequency jumps), thus outputting a lower attention value. Therefore, although It contains noise, but because The noise will not be injected due to the suppression effect. Secondly, consider the case where RGB images contain shadows. At the shadow boundaries, the RGB differences are significant, resulting in higher resolution. At this point, we inject RGB features into the depth branch. Since the depth map itself is smooth (physical plane) in the shadow areas, the texture features provided by RGB can help the depth branch refine the road surface segmentation boundaries, preventing the depth branch from becoming blurry due to a lack of texture.

[0156] (4) Bidirectional Context Stream Alignment Module (BCFA):

[0157] like Figure 4 As shown, the BCFA module is located in the decoder and aims to solve the "semantic-detail" misalignment problem in the feature recovery process from the encoder to the decoder. The multi-scale fused features processed by the CDFM module are input into the decoder module. The decoder adopts a bottom-up, hierarchical reconstruction path, starting from the deepest layer, passing through a series of upsampling units and the Bidirectional Context Flow Alignment (BCFA) module, to gradually recover the spatial resolution of the feature map.

[0158] The specific interaction mechanism of BCFA is as follows:

[0159] ① Generating space masks from top to bottom:

[0160] First, deep features shallow features The splicing is performed along the channel dimension, first passing through a... The convolutional layer performs channel dimensionality reduction and preliminary feature fusion, followed by dilated convolution to expand the receptive field, and then generates a spatial mask after sigmoid activation. This mask uses deep semantic information to indicate the spatial location of foreground objects.

[0161]

[0162] ② Bottom-up feedback channel weights:

[0163] Using the generated shallow features Masking is performed to retain features only from the foreground region. Then, global average pooling (GAP) and multilayer perceptron (MLP) are used to generate channel attention vectors. This vector reflects the importance of each feature channel within the critical region.

[0164]

[0165] Shallow features often contain a lot of irrelevant background texture (such as noise in leaves or fine cracks in the road surface). By multiplying by We used deep semantics to forcibly suppress the response of non-target regions in shallow features, which is equivalent to performing a semantic-based spatial denoising.

[0166] ③ Bidirectional feature calibration:

[0167] The calculated weights are applied back to their respective input paths to achieve deep semantic channel activation and shallow spatial noise filtering, ultimately outputting the calibrated features. The calculation process is shown in equation (5):

[0168]

[0169] This is a form of reverse guidance. For example, if shallow features detect obvious "wheel texture," the generated features will enhance the channel response for the "vehicle" category in the deep features, while suppressing the response for the "pedestrian" category. This allows deep semantics to maintain its abstractness while possessing sensitivity to the texture of specific instances.

[0170] (5) Generation of prediction head and final result:

[0171] like Figure 2 As shown in the upper right corner of the model architecture diagram, after the decoder outputs the last layer of features, the prediction and reconstruction steps are performed, which specifically include segmentation projection and final upsampling.

[0172] Segmentation projection: through a convolution kernel of size ... The convolutional layer is used as the prediction head, mapping the number of feature channels in the final output of BCFA to the number of semantic categories. This yields a preliminary semantic probability map.

[0173] Final upsampling: Since the output resolution of the prediction head is still the same as the original image... To obtain pixel-level prediction results consistent with the input image, a bilinear interpolation algorithm is used to enlarge the semantic probability map by 4 times, restoring it to the original image size. This leads to the final semantic segmentation result of the road environment.

[0174] (6) Loss function construction:

[0175] To optimize model parameters, ensure effective learning of intermediate layer features, and enhance the model's ability to perceive object boundaries, this embodiment employs a multi-task deep supervision strategy. The total loss function consists of the main segmentation loss. Auxiliary segmentation loss and boundary consistency loss The three-part weighted construction is shown in Equation (6):

[0176]

[0177] in, and The balance coefficient (in this embodiment, it is taken as...) The specific definitions of each loss function are as follows:

[0178] Main segmentation loss ( The standard cross-entropy loss is calculated at the prediction plot of the final model output, using the following formula:

[0179]

[0180] Auxiliary segmentation loss ( To add an auxiliary classification head to the deeper layers of the network, forcing the deep network to possess semantic discriminative ability early in feature propagation, effectively alleviating the gradient vanishing problem and accelerating model convergence, an auxiliary classification head is added at the output of the penultimate bidirectional context flow alignment module of the decoder, i.e., at an image resolution of [missing information]. At this point, cross-entropy loss is used for calculation.

[0181] Boundary consistency loss To verify the effectiveness of the CDFM module in repairing RGB boundaries using depth maps in this embodiment, explicit boundary constraints are introduced. However, in actual engineering, due to the extremely small number of edge pixels, directly calculating IoU leads to an excessively small denominator and unstable values. This embodiment improves upon this by introducing a weighted binary cross-entropy combined with IoU: First, the Laplacian operator is used to calculate the IoU from the predicted semantic map. and real labels Extracting edge probability maps and Binary cross-entropy loss:

[0182] (8)

[0183] in This is the positive and negative sample balance coefficient. Edge pixels are usually much fewer than non-edge pixels, so we take... .

[0184] Then the cross-union ratio (IoU) loss between the two is calculated, as shown in equation (9):

[0185]

[0186] Total marginal loss This loss function forces the object contours output by the model to maintain a high degree of consistency with the real contours, significantly improving the sharpness of the segmentation edges.

[0187] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental models, characterized in that, Includes the following steps: Step S1: Complete the semantic segmentation model construction and train it using a public dataset; Step S2: Obtain the RGB image and corresponding depth image of the driving scene as input to the model; Step S3: Input the RGB and HHA-encoded depth maps into the frozen DINOv3 backbone network and extract multi-scale general semantic features respectively; Step S4: Enter the complementary difference fusion module: calculate high-frequency difference features and generate texture and geometric attention maps; Step S5: Perform bidirectional cross-injection: geometrically enhance RGB texture, refine texture depth geometry, and output blended features; Step S6: Enter the bidirectional context flow alignment module: perform closed-loop mutual refinement using deep spatial mask and shallow channel feedback; Step S7: Input the optimized multi-scale features into the decoder for step-by-step upsampling and feature recovery; Step S8: Output pixel-level semantic classification results to generate the final road environment semantic segmentation results.

2. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 1, characterized in that, Step S3 includes the following steps: Step S31: Using the HHA encoding strategy, the depth map is converted into a three-channel geometric image; the three channels are: channel 1, representing horizontal parallax, used to simulate parallax information in binocular vision, which is extremely sensitive to object edges; channel 2, representing ground height, used to represent height information after calibration using the gravity direction; and channel 3, representing the angle of the normal vector, used to represent the angle between the surface normal vector and the gravity direction. The geometric image after HHA encoding is aligned with the RGB image in terms of data dimension, and in terms of information content, the implicit distance information is made explicit as geometric structure information.

3. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 2, characterized in that, Step S3 further includes the following steps: Step S32: Input the RGB image and the HHA encoded geometric image into two DINOv3 networks with the same structure but independent from each other. During the entire training process, all parameters of DINOv3 are kept frozen, that is, the gradient is not backpropagated. Then, the output of the 3rd, 6th, 9th and 12th layers of DINOv3 are extracted as the input of the feature pyramid. Then, the resolution is adjusted to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image by bilinear interpolation.

4. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: Perform modality-specific preprocessing on the reconstructed features from step S3: RGB branch uses... Convolution performs channel mapping, preserving global semantics, denoted as The HHA branch adopts Depthwise separable convolution is used to enhance geometric edges, denoted as... ; Step S42: In each level of the encoder, in order to solve the problem of noise contamination between modes, a complementary difference fusion module is introduced to perform lateral feature fusion. This module first calculates high-frequency difference features, defining kernel sizes as follows: and Average pooling operation; extracting high-frequency details by subtracting aggregated context features from the original features. Taking the RGB branch as an example, differential features... The calculation is shown in the following formula: ； It can respond to extremely subtle texture changes, but also includes most of the sensor speckle noise; The response to structural edges at a larger scale has a certain smoothing and suppression effect on isolated noise points. This method concatenates the two in the channel dimension to construct a multi-scale frequency domain descriptor. The subsequent convolutional network will use this descriptor to determine whether a high-frequency response belongs to "structural edge" or "random noise". Step S43: Generate a texture attention map using convolution and the Sigmoid function. and geometric attention graph .

5. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 1, characterized in that, Step S5 includes the following steps: Perform bidirectional cross-injection: leveraging geometric attention Injecting HHA geometric features into the RGB branch to fix shadow blur; utilizing texture attention Instruct RGB feature injection into the HHA branch to suppress edge noise; ultimately Fusion features The calculation is shown in the following formula: ； In the formula, the convolutional layer in fused is It is a learnable convolutional layer. , The scaling factor is learnable; RGB features are injected into the depth branch. Since the depth map itself is smooth in the shadows, the texture features provided by RGB can help the depth branch refine the road surface segmentation boundary and prevent the segmentation from becoming blurry due to the lack of texture in the depth branch.

6. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 1, characterized in that, Step S6 includes the following steps: The multi-scale fused features, processed by the bidirectional context flow alignment module, are input into the decoder module. The decoder adopts a bottom-up, step-by-step reconstruction path, starting from the deepest layer and gradually restoring the spatial resolution of the feature map through a series of upsampling units and the bidirectional context flow alignment module. The specific interaction mechanism is as follows: Step S61: Generate spatial mask from top to bottom: First, generate deep features shallow features The splicing is performed along the channel dimension, first passing through a... The convolutional layer performs channel dimensionality reduction and preliminary feature fusion, followed by dilated convolution to expand the receptive field, and then generates a spatial mask after sigmoid activation. This mask uses deep semantic information to indicate the spatial location of the foreground object, as shown in the following formula: ； Step S62: Bottom-up feedback channel weights: using the generated shallow features Masking is performed to retain only the features of the foreground region. Then, global average pooling and multilayer perceptron processing are used to generate channel attention vectors. This vector reflects the importance of each feature channel within the key region, as shown in the following formula: ； Shallow features often contain a lot of irrelevant background textures, which this method addresses by multiplying by... It also uses deep semantics to forcibly suppress the response of non-target regions in shallow features, which is equivalent to performing a semantic-based spatial denoising. Step S63: Bidirectional Feature Calibration: The calculated weights are applied back to their respective input paths to achieve deep semantic channel activation and shallow spatial noise filtering, ultimately outputting the calibrated features. The calculation process is shown in the following formula: ； In the formula It is a reverse guidance; this allows deep semantics to maintain its abstractness while being sensitive to the texture of specific instances.

7. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 1, characterized in that, Step S7 includes the following steps: After the decoder outputs the last layer of features, a prediction and reconstruction step is performed, including segmentation projection and final upsampling, as follows: Step S71: Segment the projection, using a convolution kernel of size... The convolutional layer is used as the prediction head, mapping the number of feature channels in the final output of the bidirectional context flow alignment module to the number of semantic categories. This yields a preliminary semantic probability map; Step S72: Final upsampling, since the output resolution of the prediction head is still the same as the original image. To obtain pixel-level prediction results consistent with the input image, a bilinear interpolation algorithm is used to enlarge the semantic probability map by 4 times, restoring it to the original image size. This leads to the final semantic segmentation result of the road environment.

8. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 1, characterized in that, The semantic segmentation method also includes the construction of a loss function: To optimize model parameters, ensure effective learning of intermediate layer features, and enhance the model's ability to perceive object boundaries, this method employs a multi-task deep supervision strategy: the total loss function is composed of the main segmentation loss. Auxiliary segmentation loss and boundary consistency loss The calculation formula is as follows: The three-part weighted construction is shown in the following equation: ； In the formula and This is the balance coefficient.

9. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 8, characterized in that, The construction of the loss function also includes the following steps: The main segmentation loss The standard cross-entropy loss is calculated at the prediction plot of the final model output, as shown in the following formula: ； The auxiliary segmentation loss To add an auxiliary classification head to deeper layers of the network, forcing the deep network to possess semantic discriminative ability early in feature propagation, effectively alleviating the gradient vanishing problem and accelerating model convergence, an auxiliary classification head is added at the output of the penultimate bidirectional context flow alignment module of the decoder, i.e., at an image resolution of [missing information]. At this point, cross-entropy loss is used for calculation.

10. The road environment semantic segmentation method based on cross-modal differential modulation and visual fundamental model according to claim 8, characterized in that, The construction of the loss function also includes the following steps: Boundary consistency loss Introducing weighted binary cross-entropy combined with IoU: First, the Laplacian operator is used to analyze the predicted semantic graph. and real labels Extracting edge probability maps and Binary cross-entropy loss: （8）； In the formula The positive and negative sample balance coefficient is used, where edge pixels are much fewer than non-edge pixels. ; Then the cross-union ratio (IoU) loss between the two is calculated, as shown in the following formula: ； Total marginal loss This loss function forces the object contours output by the model to be highly consistent with the real contours, significantly improving the sharpness of the segmentation edges.

Images