A road elevation reconstruction method based on text semantic guidance and feature decoupling

Abstract

The application provides a road surface reconstruction method based on a direction perception pseudo binocular network, comprising a direction perception feature enhancement module: by combining road geometric features with view angle changes, multiple feature enhancement mechanisms such as a spatial saliency perception unit and an internal direction decoupling module are adopted, so that the network can effectively enhance the perception ability of the road surface geometry; a pseudo binocular cost volume construction operator is proposed: by introducing feature difference modeling and a nonlinear gating mechanism, the parallax effect in binocular vision is simulated, and a pseudo binocular disparity volume is efficiently constructed under the condition of monocular input. Therefore, the application first realizes high-precision and high-robustness three-dimensional road reconstruction under the condition of monocular input, and the innovation lies in the introduction of direction perception and pseudo binocular disparity modeling technology, which is especially suitable for complex and variable road scenes. In practical applications, the technology can be widely used in the fields of automatic driving, high-precision map construction and the like, and has wide market prospects and technical value.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of transportation and computer vision, and more particularly to a road surface reconstruction method based on a direction-aware pseudo-binocular network. Background Technology

[0002] With the rapid development of intelligent transportation systems and autonomous driving technologies, the demand for high-precision, real-time perception and reconstruction of road environments, especially road surface geometry, is becoming increasingly urgent. In existing technologies, road reconstruction mainly relies on LiDAR, stereo vision, or monocular vision-based depth estimation methods. LiDAR can directly acquire high-precision 3D point clouds, but its equipment is expensive, deployment is complex, data is sparse, and it is easily affected by weather, making large-scale deployment and application difficult. Stereo vision-based methods calculate disparity by matching corresponding points in the left and right views, thereby recovering elevation information. However, its performance heavily depends on high-quality calibration, sufficient texture information, and good lighting conditions. In areas with large areas of weak texture on the road surface (such as flat asphalt pavement) or in scenarios with repetitive structures, matching is relatively difficult, easily leading to reconstruction failure or a significant decrease in accuracy.

[0003] In recent years, depth estimation techniques based on monocular vision, especially deep learning-based methods, have attracted widespread attention due to their single-camera requirement, low cost, and ease of deployment. However, in complex road scenarios (such as those with water accumulation and reflection, shadows, damage, severe wear, or extreme lighting changes), the predicted depth maps are prone to problems such as missing structures, distortion, or excessive noise. In addition, most methods lack explicit modeling of the directional structure of the road surface (such as driving direction, curb orientation, lane extension direction, etc.), while these directional cues are crucial for improving the geometric consistency and structural integrity of road reconstruction. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a road elevation reconstruction method based on textual semantic guidance and feature decoupling. This method constructs a pseudo-binocular vision network by introducing cross-modal fusion and direction-aware mechanisms, achieving high-precision joint estimation of road surface elevation, texture, and damage information. It aims to provide real-time and accurate 3D road surface data support for autonomous driving, road maintenance, and digital twin traffic.

[0005] This invention is achieved through the following technical solution: a road surface reconstruction method based on a direction-aware pseudo-binocular network, specifically including the following steps: Step 1: Acquisition and preprocessing of road surface data based on the RSRD dataset, specifically including the following steps: Step 1-1: Select road scene data from the RSRD dataset and obtain raw multi-sensor data containing the true values ​​of the real road surface elevation GT; Steps 1-2: Spatial cropping and filtering of the collected laser point cloud data; Steps 1-3: Perform spatial consistency processing on the synchronously acquired camera images: The image is remapped to a regular 164×64 grid structure with a frontal field of view consistent with the point cloud. Steps 1-4: Divide the dataset into training and testing sets: The training and test sets are divided into 70% and 30% sets according to the time series sequence. Step 2: Construction and projection of BEV space based on multi-scale features: Step 2-1: Establish the road reference coordinate system and BEV spatial representation: Representing the three-dimensional road surface in... XYZ In a spatial coordinate system with reference to, where XZ The plane corresponds to the horizontal ground. Y The axis represents the vertical direction; based on this coordinate system, the road surface is discretized into a BEV mesh on the horizontal plane, and a BEV mesh set is constructed, represented as... (1) make ,Right now To cover the horizontal projection area of ​​the road N Each grid point has a grid point. Corresponding to a unique location in the BEV space, It is a binary mask used to mark whether grid points contain valid information; Step 2-2: Multi-scale feature extraction based on EfficientNet-B6 network. The specific steps are as follows: Step 2-2-1: Input road image as follows ,in , The original image resolution is used to align it to a spatial coordinate system and input it into the EfficientNet-B6 backbone network to extract multi-scale features, represented as follows: (2) in This represents the pre-trained EfficientNet-B6 network. , {1,2,3} , indicating the first k The output feature map of the layer, , and These represent the number of channels, height, and width of the feature map, respectively. Step 2-2-2: Flatten the feature map along the spatial dimensions as follows: (3) This transformation ensures that each subsequent BEV grid point is mapped to its corresponding image projection coordinates. In this process, index queries are performed to achieve cross-domain feature sampling from the image domain to the BEV space; Steps 2-3, BEV space feature mapping based on projection index, specifically include the following steps: The three-dimensional coordinates of the center point of each voxel in the BEV mesh are represented as follows: . ; Step 2-3-1: Based on the camera's extrinsic and intrinsic parameters, convert the three-dimensional coordinates... Projecting onto the image plane yields its corresponding two-dimensional pixel coordinates, denoted as... ; Step 2-3-2, Calculation In the flattened feature map Linear indexes in: (4) in The width of the feature map is given, and the indices of all valid projection points are combined into a single value. ; Step 2-3-3: Reconstruction of 3D Voxel Features Using index-based tensor sampling operations ( gather )exist The image features corresponding to each voxel are retrieved in the image and represented as follows: (5) The sampled features were then re-sampled. reshape For the three-dimensional structure of BEV: (5) in Z,X These are the length and width of the front area of ​​the vehicle, respectively. Y It is an estimated elevation; Step 3: BEV feature enhancement based on text semantic guidance: Step 3-1: Descriptive text generation based on a lightweight multimodal large language model: Step 3-1-1: Introduce a set of structured cue sets P to constrain the output content of the multimodal generative model; the cue set P is defined as a set of semantic cue templates, and its mathematical expression is: ,...(6) Each prompt corresponds to a road semantic attribute description dimension, as shown in the following example: =“Please describe the trend of longitudinal or transverse slope changes in the road surface”; =“Please describe whether there are any potholes, cracks, or damaged areas, and explain their distribution”; =“Please describe whether there are areas of water accumulation or mirror-like reflections”; =“Please describe the visibility and integrity of lane lines, curbs, or other road structure markings”; Step 3-1-2: Obtain the road surface image And the prompt set P as a common input to the multimodal generative model Generate corresponding descriptive text T Its generation process can be formally represented as: T= ( , P), (7) in , is a natural language sequence consisting of several words or sub-words, where L represents the length of the text sequence; Step 3-1-3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation Input to pre-trained text encoding model The preferred text encoding model is BERT or its improved version, used for context-sensitive deep semantic modeling of text sequences, i.e. , where serialized vector representation Each vector Indicates the first in the text i The semantic embedding of a word in a global context. For text feature dimensions; Step 3-2, BEV feature dimension transformation: The four-dimensional BEV features obtained in steps 2-3 are flattened and rearranged along the spatial and elevation dimensions; the Z, X, and Y dimensions are merged into a single sequence dimension, thereby converting the BEV features into a two-dimensional feature sequence representation. ,in , ; Step 3-3: Cross-attention calculation: Multi-scale BEV features As a query vector, Query uses textual semantic features As a key vector (Key) and a value vector (Value), it is represented as follows: , .. (8) in , and These are the learnable linear mapping matrices, d This is the scaling factor in the attention mechanism. This represents the BEV space feature representation enhanced by textual semantic features; Step 3-3-1, Nonlinear Mapping and Residual Connection: Output BEV features Apply feedforward network and residual connection: (9) in The projection matrix used for dimension matching. The representation layer is normalized, and then the features are reshaped into a BEV 3D structure. Step 4, Design of the pseudo-binocular fusion module: Since real-world road reconstruction scenarios typically only provide monocular images or sparse depth information, directly constructing traditional binocular disparity volumes has objective limitations. This invention proposes a pseudo-auxiliary view generation method based on feature space translation and geometric consistency constraints. By constructing multiple "hypothetical viewpoints" within the BEV feature domain, it simulates feature correspondences under different disparity assumptions. Step 4-1, Main View Feature Definition: Define the multimodal augmentation feature set output in Step 3. Perform a copy operation and define it as a main view feature, i.e. . ; Step 4-2, Pseudo-auxiliary feature generation: Step 4-2-1, Spatial Saliency Perception Unit: First, Channel adaptation is performed using 1×1 convolution, followed by extraction of local geometric continuity in the BEV mesh using 3×3 convolution: (10) Then, a saliency-enhanced output is generated using a standard spatial attention mechanism, as follows: (11) in, They represent in The output responses of global average pooling and global max pooling operations performed on the feature map are concatenated along the channel dimension and then further reduced in dimensionality using a 3×3 convolution operation. This is then processed by... The sigmoid activation function is used to generate a saliency weight map, and the results are compared with... Performing a dot product operation yields the final spatial saliency-enhancing features. ; Step 4-2-2, Decoupling the intrinsic direction of lightweighting: First, in order to map the features to an orientation-sensitive, separable intermediate subspace, a channel-compressed convolution is performed, represented as: ( (8) in Indicates the normalization layer; Then, decoupling is performed in the horizontal and vertical directions respectively: (9) (10) in, Convolutional layer representation only uses The strip-shaped core slides in the height direction. Then it means to use The transverse strip core slides in the width direction; finally, using 3 Convolution fuses long-range responses and local compensations within a two-dimensional local area, generating directional and spatially consistent feature maps. and ; Step 4-2-3, Structural splicing and one-dimensional convolutional recombination: The above three types of features are concatenated according to the channel dimension: (11) Enhance features Reorganize the BEV network location into a serialized representation; first, flatten it in the spatial dimension: (16) in The number of channels output by the hybrid convolutional stream is given; then, one-dimensional convolution is used to perform context aggregation and structural reorganization on the serialized features: (17) in This represents a one-dimensional convolution operator performed along the sequence dimension; finally, it is reshaped back to the BEV space structure, and its output is defined as a pseudo-auxiliary view feature. ; Step 4-3: Construction of the learnable cost body: Obtaining main view features and pseudo-auxiliary features Subsequently, a learnable cost volume construction operator for monocular BEV scenarios is proposed. This operator explicitly constructs a pseudo-feature cost volume with geometric discrimination capability by introducing feature difference modeling, nonlinear gated attention, and an adaptive offset mechanism inspired by stereo parallax, without real second-view input. Step 5: Road elevation prediction and 3D reconstruction output: After completing the construction of the pseudo-feature cost volume, based on a reliable elevation regression head, the feature representation in the BEV space is used to predict the elevation at each location, and the three-dimensional geometry of the road is recovered accordingly.

[0006] As a preferred option, step 1-2 specifically includes the following steps: Step 1-2-1: Clip the spatial range of the 32-beam lidar point cloud data along the X and Y axes of the vehicle coordinate system. The clipping intervals are set to [-1.0m, 0.9m] and [2.1m, 7.1m], respectively, retaining the data of the effective road surface area in front of the road. Step 1-2-2: Divide the cropped irregular point cloud into regular grid units with a size of 3cm×3cm; Steps 1-2-3: Use the average elevation value of all points in the grid as the GT elevation corresponding to the grid, and set a binary mask to record the missing areas of no point data to obtain a regularized road surface elevation ground value raster.

[0007] As a preferred option, step 4-3 specifically includes the following steps: Step 4-3-1, Feature difference modeling of main and secondary views: Element-wise difference operations are performed on the main view features and pseudo-auxiliary features within the BEV space to explicitly characterize their inconsistencies in spatial structure and geometric distribution. That is: (18) Step 4-3-2, Mask generation based on nonlinear gating: First, for difference features Perform a global max pooling operation to compress its spatial dimensionality and extract the most salient response information to obtain a global description vector. Subsequently, the pooled feature vectors are input into a nonlinear mapping module consisting of two multilayer perceptrons to achieve feature selection and gating modeling. This process is formally expressed as: (19) in, , This represents the learnable weight matrix. ReLU represents the nonlinear activation function of the intermediate layer. express sigmoidActivation function, output A normalized dual-channel weight mask, used for subsequent feature weighting and offset modeling; Step 4-3-3, Adaptive Shift and Feature Weighting: mask m The two channels correspond to the weighting coefficients of the main view features and the pseudo-auxiliary features, respectively: (20) Then, the features are reweighted through element-wise multiplication to obtain the enhanced feature representation: = , = , (twenty one); Step 4-3-4, Pseudo-binocular feature fusion: The weighted main view features and pseudo-auxiliary features are concatenated along the channel dimension to form a local geometric encoding representation: , (twenty two) Should It is defined as the pseudo-feature cost volume in a monocular BEV scenario.

[0008] As a preferred option, step 5 specifically includes the following steps: Step 5-1, Feature encoding of the elevation regression head: To model the local-global elevation correlation within the BEV space, an elevation regression network commonly used in road reconstruction methods is introduced to perform layer-by-layer feature transformation on the cost volume. , (twenty three) in, This represents a context encoding function composed of multiple convolutional layers and normalized units. This is a mid-level elevation-sensing feature representation used to regress the elevation value for each BEV grid; Step 5-2, Road Elevation Continuous Regression Prediction: By introducing a lightweight elevation regression head Continuous elevation prediction is performed for each BEV grid location, and this is done using the binary mask constructed in step 2. Constraints are applied to the output to ensure that the prediction results are only effective within road areas where real-world supervision exists, thus obtaining valid predicted road elevation results. It is expressed as follows: , (twenty four) Step 5-3, Output of 3D Road Surface Reconstruction: Based on the definition of the BEV mesh's position in the spatial coordinate system, the predicted elevation is mapped back to three-dimensional space to obtain a three-dimensional point set representation of the road surface: in, The coordinates of the BEV mesh in the horizontal plane; The final output is the 3D road reconstruction result; Step 5-4, Loss Function: During the training phase, to constrain the consistency between the predicted elevation and the actual elevation, the supervised elevation regression loss, which is widely used in existing road elevation regression tasks, is adopted: in Marking the actual road elevation. Indicates the number of valid grid cells.

[0009] This invention, by employing the above technical solutions, offers the following advantages compared to existing technologies: First, addressing the challenges of weak texture, small gradients, and difficulty in stably extracting geometric structures in distant road regions, this invention proposes a direction-aware feature enhancement module. Based on the directional features of the road structure, this module enhances the network's ability to express sparse and geometrically invariant features by constructing a direction context-aware mechanism, improving the geometric consistency and stability of deep features, thereby effectively alleviating the structural degradation problem caused by perspective changes in distant regions. Second, this invention proposes a pseudo-binocular stereo fusion module to solve the problem of traditional reconstruction algorithms struggling to capture global dependencies in low-texture and distant scenes. This module establishes long-range cross-scale associations in the spatial and channel dimensions through a pseudo-stereo feature generation strategy, fully compensating for reconstruction biases caused by weak surface features and information imbalances in distant roads. It is worth noting that all modules proposed in this invention are designed with lightweight principles, allowing them to be embedded as plug-and-play components into any road reconstruction network. Therefore, by introducing a direction perception and pseudo-stereo fusion mechanism, this invention overcomes the shortcomings of traditional methods such as elevation instability, structural distortion, and large computational burden in long-distance areas, and achieves high-precision and high-efficiency three-dimensional reconstruction of road surfaces in autonomous driving scenarios. It has good engineering applicability and promotion value.

[0010] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0011] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is an example of a data sample for the present invention; Figure 2 This is a diagram illustrating the overall framework of the network model. Detailed Implementation

[0012] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0013] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0014] The following is combined with Figures 1 to 2 The road reconstruction method based on a direction-aware pseudo-binocular network according to an embodiment of the present invention will be described in detail.

[0015] This invention proposes a road surface reconstruction method based on a direction-aware pseudo-binocular network, which specifically includes the following steps: Step 1: Acquisition and preprocessing of road surface data based on the RSRD dataset, specifically including the following steps: Step 1-1: Select road scene data from the RSRD dataset according to the official process and obtain raw multi-sensor data containing the true value GT of the real road surface elevation. Steps 1-2: Spatial cropping and filtering of the collected laser point cloud data; specifically including the following steps: Step 1-2-1: Clip the spatial range of the 32-beam lidar point cloud data along the X and Y axes of the vehicle coordinate system. The clipping intervals are set to [-1.0m, 0.9m] and [2.1m, 7.1m], respectively, retaining the data of the effective road surface area in front of the road. Step 1-2-2: Divide the cropped irregular point cloud into regular grid cells of size 3cm×3cm for grouping and indexing; Steps 1-2-3: Use the average elevation value of all points in the grid as the GT elevation corresponding to the grid, and set a binary mask to record the missing areas of no point data to obtain a regularized road surface elevation ground value raster.

[0016] Steps 1-3: Perform spatial consistency processing on the synchronously acquired camera images: The image is remapped to a regular 164×64 grid structure with the same frontal field of view as the point cloud, ensuring the uniformity of the image and the point cloud in spatial distribution and scale, and providing a consistent input format for subsequent feature alignment and network training. Steps 1-4: Divide the dataset into training and testing sets: The training set and the test set are divided into 70% and 30% according to the time series order to ensure that there is no spatial overlap between the training set and the test set. Step 2: Construction and projection of BEV space based on multi-scale features: Step 2-1: Establish the road reference coordinate system and BEV spatial representation: Representing the three-dimensional road surface in... XYZ In a spatial coordinate system with reference to, where XZ The plane corresponds to the horizontal ground. Y The axis represents the vertical direction; based on this coordinate system, the road surface is discretized into a BEV (bird's-eye view) grid on the horizontal plane, and a BEV grid set is constructed, represented as... (1) make ,Right now To cover the horizontal projection area of ​​the road N Each grid point has a grid point. Corresponding to a unique location in the BEV space, It is a binary mask used to mark whether grid points contain valid information; Step 2-2: Multi-scale feature extraction based on EfficientNet-B6 network. The specific steps are as follows: Step 2-2-1: Input road image as follows ,in , The original image resolution is used to align it to a spatial coordinate system and input it into the EfficientNet-B6 backbone network to extract multi-scale features, represented as follows: (2) in This represents the pre-trained EfficientNet-B6 network. , {1,2,3} , indicating the first k The output feature map of the layer, , and These represent the number of channels, height, and width of the feature map, respectively. Step 2-2-2: To facilitate subsequent point-level sampling based on projected coordinates, the feature map is flattened along the spatial dimension as follows: (3) This transformation allows each subsequent BEV grid point (or voxel) to be mapped to its corresponding image projection coordinates. In this process, index queries are performed to achieve cross-domain feature sampling from the image domain to the BEV space; Steps 2-3, BEV space feature mapping based on projection index, specifically include the following steps: The three-dimensional coordinates of the center point of each voxel in the BEV mesh are represented as follows: . ; Step 2-3-1: Based on the camera's extrinsic and intrinsic parameters, convert the three-dimensional coordinates... Projecting onto the image plane yields its corresponding two-dimensional pixel coordinates, denoted as... ; Step 2-3-2, Calculation In the flattened feature map Linear indexes in: (4) in The width of the feature map is given, and the indices of all valid projection points are combined into a single value. ; Step 2-3-3: Reconstruction of 3D Voxel Features Using index-based tensor sampling operations ( gather )exist The image features corresponding to each voxel are retrieved in the image and represented as follows: (5) The sampled features were then re-sampled. reshape For the three-dimensional structure of BEV: (5) in Z,X These are the length and width of the front area of ​​the vehicle, respectively. Y It is an estimated elevation; Step 3: BEV Feature Enhancement Based on Textual Semantic Guidance To enhance the ability of BEV features to express high-level semantic information of road surface, this invention proposes a cross-modal feature fusion method based on a cross-attention mechanism. This method guides and modulates BEV spatial features through textual semantic features, thereby obtaining semantically enhanced BEV feature representations.

[0017] Step 3-1: Descriptive text generation based on a lightweight multimodal large language model: To compensate for the inadequacy of purely visual features in modeling implicit attributes such as slope and reflectivity, this invention introduces explicit road semantic priors generated by a large model. This prior can generate natural language descriptive text consistent with the semantics of the image content under visual input conditions. In this embodiment, the multimodal generation model can be selected from BLIP-2, LLaVA, GPT-4V, or equivalent models, but this invention does not limit the specific model structure. Specifically, it includes the following steps: Step 3-1-1: To ensure that the generated text stably and comprehensively covers the key semantic information of the road surface, this invention introduces a set of structured prompts P to constrain the output content of the multimodal generation model. The prompt set P is defined as a set of semantic prompt templates, and its mathematical expression is: ,...(6) Each prompt corresponds to a road semantic attribute description dimension, as shown in the following example: =“Please describe the trend of longitudinal or transverse slope changes in the road surface”; =“Please describe whether there are any potholes, cracks, or damaged areas, and explain their distribution”; =“Please describe whether there are areas of water accumulation or mirror-like reflections”; =“Please describe the visibility and integrity of lane lines, curbs, or other road structure markings”; Step 3-1-2: Obtain the road surface image And the prompt set P as a common input to the multimodal generative model Generate corresponding descriptive text T Its generation process can be formally represented as: T= ( , P), (7) in , is a natural language sequence consisting of several words or sub-words, where L represents the length of the text sequence.

[0018] Step 3-1-3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation Input to pre-trained text encoding model In this invention, the preferred text encoding model is BERT or an improved version thereof, used for context-sensitive deep semantic modeling of text sequences. , where serialized vector representation Each vector Indicates the first in the text i The semantic embedding of a word in a global context. For text feature dimensions.

[0019] Step 3-2: BEV Feature Dimension Transformation: To facilitate subsequent attention-based multimodal fusion processing, this invention flattens and rearranges the four-dimensional BEV features obtained in steps 2-3 along the spatial and elevation dimensions. The Z, X, and Y dimensions are merged into a single sequence dimension, thereby converting the BEV features into a two-dimensional feature sequence representation. ,in , .

[0020] Step 3-3: Cross-attention calculation: Multi-scale BEV features As a query vector, using textual semantic features As a key vector and a value vector, it is represented as: , .. (8) in , and These are the learnable linear mapping matrices, d This is the scaling factor in the attention mechanism. This represents the BEV space feature representation enhanced by textual semantic features.

[0021] Step 3-3-1: Nonlinear mapping and residual connection: To further enhance expressive power while ensuring stability, this patent addresses the output BEV characteristics. Apply feedforward network and residual connection: (9) in The projection matrix used for dimension matching. The representation layer is normalized, and then the features are reshaped into a BEV 3D structure.

[0022] Step 4, Design of the pseudo-binocular fusion module: Since real-world road reconstruction scenarios typically only provide monocular images or sparse depth information, directly constructing traditional binocular disparity volumes has objective limitations. This invention proposes a pseudo-auxiliary view generation method based on feature space translation and geometric consistency constraints. By constructing multiple "hypothetical viewpoints" within the BEV feature domain, it simulates feature correspondences under different disparity assumptions. Step 4-1, Main View Feature Definition: Define the multimodal augmentation feature set output in Step 3. Perform a copy operation and define it as a main view feature, i.e. . ; Step 4-2, Pseudo-auxiliary feature generation: This step is used to identify key regions in the BEV feature space, and their saliency reflects the importance of the actual geometry. Specifically: Step 4-2-1: Spatial saliency sensing unit: First, Channel adaptation is performed using 1×1 convolution, followed by extraction of local geometric continuity in the BEV mesh using 3×3 convolution: (10) Then, a saliency-enhanced output is generated using a standard spatial attention mechanism, as follows: (11) in, They represent in The output responses of global average pooling and global max pooling operations performed on the feature map are concatenated along the channel dimension and then further reduced in dimensionality using a 3×3 convolution operation. This is then processed by... The sigmoid activation function is used to generate a saliency weight map, and the results are compared with... Performing a dot product operation yields the final spatial saliency-enhancing features. ; Step 4-2-2: Decoupling the intrinsic direction of lightweighting: This invention provides a lightweight convolutional module with "vertical-lateral sequential decoupling" without adding extra sensors, aiming to significantly improve the model's sensitivity to elevation morphology. Specifically: First, in order to map the features to an orientation-sensitive, separable intermediate subspace, a channel-compressed convolution is performed, represented as: ( (8) in Indicates the normalization layer; Then, decoupling is performed in the horizontal and vertical directions respectively: (9) (10) in, Convolutional layer representation only uses The strip-shaped core slides in the height direction. Then it means to use The transverse strip core slides in the width direction; finally, using 3 Convolution fuses long-range responses and local compensations within a two-dimensional local area, generating directional and spatially consistent feature maps. and ; Step 4-2-3, Structural splicing and one-dimensional convolutional recombination: The above three types of features are concatenated along the channel dimension to form a holistic feature field containing multi-structural information about the road, as follows: (11) To further integrate information on the "parallax candidate dimension", the enhanced features will be... Reorganize the BEV network into a serialized representation based on its location. Specifically, first, flatten it in the spatial dimension: (16) in This represents the number of channels output by the hybrid convolutional stream. Then, one-dimensional convolution is used to perform context aggregation and structural reorganization on the serialized features: (17) in This represents a one-dimensional convolution operator performed along the sequence dimension. This operation can be viewed as modeling cross-regional dependencies of features at different locations in the BEV space, thereby enhancing the disparity sensitivity of the features. Finally, it is reshaped back to the BEV space structure, and its output is defined as a pseudo-auxiliary view feature. .

[0023] Step 4-2-4, Output and Property Description: (1) The above operations are not only sensitive to the geometric structure of slope and continuous elevation changes, but also make the number of parameters and calculations much smaller than those of the standard 3 with the same receptive field. 3 convolutions reduced the number of cells by 33%.

[0024] (2) Output and The longitudinal trend and lateral details are encoded separately, both of which are aligned with the original BEV space (same size) and can be directly used for subsequent reconstruction; (3) Pseudo-auxiliary features generated by the above steps Features of the main view While strictly aligned in the spatial dimension, it introduces a "parallax-like difference" in the feature distribution resulting from multi-scale fusion and structural reorganization. This pseudo-auxiliary feature does not depend on the input of the real second-viewpoint, yet it forms a distinguishable matching relationship in the feature space, providing a stable and learnable auxiliary view representation for the subsequent cost volume modeling function.

[0025] Step 4-3: Construction of the learnable cost body: Obtaining main view features and pseudo-auxiliary features Furthermore, this invention proposes a learnable cost volume construction operator for monocular BEV scenarios. This operator, by introducing feature difference modeling, nonlinear gated attention, and an adaptive offset mechanism inspired by stereo parallax, explicitly constructs a pseudo-feature cost volume with geometric discriminative capabilities without real second-view input, providing a reliable basis for subsequent depth and elevation inference. Specifically, it includes the following steps: Step 4-3-1, Feature difference modeling of main and secondary views: Element-wise difference operations are performed on the main view features and pseudo-auxiliary features within the BEV space to explicitly characterize their inconsistencies in spatial structure and geometric distribution. That is: (18) Step 4-3-2, Mask generation based on nonlinear gating: To further from This invention selects the spatial regions most critical for depth discrimination and introduces an attention modeling method combining adaptive aggregation and nonlinear gating mechanism. First, it analyzes the differential features... Perform a global max pooling operation to compress its spatial dimensionality and extract the most salient response information to obtain a global description vector. Subsequently, the pooled feature vectors are input into a multi-layer perceptron consisting of two layers. MLP A nonlinear mapping module, consisting of [missing information], is used to achieve feature selection and gating modeling. This process is formally expressed as: (19) in, , This represents the learnable weight matrix. ReLU represents the nonlinear activation function of the intermediate layer. express sigmoid Activation function, output A normalized dual-channel weight mask, used for subsequent feature weighting and offset modeling; Step 4-3-3, Adaptive Shift and Feature Weighting: To implicitly simulate the feature shift effect caused by the second viewpoint within a monocular BEV modeling framework, this invention uses the aforementioned dual-channel weight mask. m This can be interpreted as a learnable offset and importance allocation strategy inspired by stereo parallax mechanisms. Specifically, the mask... m The two channels correspond to the weighting coefficients of the main view features and the pseudo-auxiliary features, respectively: (20) Then, the features are reweighted using the element-wise Hadamard Product to obtain the enhanced feature representation: = , = , (twenty one); The above process can be viewed as introducing a lightweight weighted projection mechanism with adaptive response capabilities into the feature space, which is equivalent to an implicitly learnable receptive field adjustment process. This mechanism automatically learns the importance distribution and response range of features through a data-driven approach, thereby effectively characterizing the potential viewpoint shift effect without explicit geometric modeling or manual setting of disparity parameters.

[0026] Step 4-3-4, Pseudo-binocular feature fusion: The weighted main view features and pseudo-auxiliary features are concatenated along the channel dimension to form a local geometric encoding representation: , (twenty two) Should It is defined as the pseudo-feature cost volume in a monocular BEV scenario.

[0027] Step 5: Road elevation prediction and 3D reconstruction output: After constructing the pseudo-feature cost volume, based on a reliable elevation head, position-by-position elevation prediction is performed on the feature representations in the BEV space, and the three-dimensional geometry of the road is reconstructed accordingly. This includes the following steps: Step 5-1, Feature encoding of the elevation regression head: To further model the local-global elevation correlation within the BEV space, this invention introduces an elevation regression network commonly used in road reconstruction methods to perform layer-by-layer feature transformation on the cost volume: , (twenty three) in, This represents a context encoding function composed of multiple convolutional layers and normalized units. The intermediate elevation-aware feature representation is used to regress the elevation value of each BEV grid; this encoding process is mainly used to fuse local contextual information and does not introduce additional structural assumptions.

[0028] Step 5-2, Road Elevation Continuous Regression Prediction: By introducing a lightweight elevation regression head Continuous elevation prediction is performed for each BEV grid location, and this is done using the binary mask constructed in step 2. Constraints are applied to the output to ensure that the prediction results are only effective within road areas where real-world supervision exists, thus obtaining valid predicted road elevation results. It is expressed as follows: , (twenty four) Step 5-3, Output of 3D Road Surface Reconstruction: Based on the definition of the BEV mesh's position in the spatial coordinate system, the predicted elevation is mapped back to three-dimensional space to obtain a three-dimensional point set representation of the road surface: in, The coordinates of the BEV mesh in the horizontal plane; The final output is the 3D road reconstruction result; Step 5-4, Loss Function: During the training phase, to constrain the consistency between the predicted elevation and the actual elevation, the supervised elevation regression loss, which is widely used in existing road elevation regression tasks, is adopted: in Marking the actual road elevation. Indicates the number of valid grid cells.

[0029] In the description of this invention, the term "a plurality of" refers to two or more. Unless otherwise explicitly defined, the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. The terms "connection," "installation," "fixing," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0030] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0031] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A road surface reconstruction method based on a direction-aware pseudo-binocular network, characterized in that, Specifically, the following steps are included: Step 1: Acquisition and preprocessing of road surface data based on the RSRD dataset, specifically including the following steps: Step 1-1: Select road scene data from the RSRD dataset and obtain raw multi-sensor data containing the true values ​​of the real road surface elevation GT. Steps 1-2: Spatial cropping and filtering of the collected laser point cloud data; Steps 1-3: Perform spatial consistency processing on the synchronously acquired camera images: The image is remapped to a regular 164×64 grid structure with a frontal field of view consistent with the point cloud. Steps 1-4: Divide the dataset into training and testing sets: The training and test sets are divided into 70% and 30% sets according to the time series sequence. Step 2: Construction and projection of BEV space based on multi-scale features: Step 2-1: Establish the road reference coordinate system and BEV spatial representation: The three-dimensional road surface is represented in... XYZ In a spatial coordinate system with reference to, where XZ The plane corresponds to the horizontal ground. Y The axis represents the vertical direction; based on this coordinate system, the road surface is discretized into a BEV mesh on the horizontal plane, and a BEV mesh set is constructed, represented as... (1) make ,Right now To cover the horizontal projection area of ​​the road N Each grid point Corresponding to a unique location in the BEV space, It is a binary mask used to mark whether grid points contain valid information; Step 2-2: Multi-scale feature extraction based on EfficientNet-B6 network. The specific steps are as follows: Step 2-2-1: Input road image as follows ,in , The original image resolution is used to align it to a spatial coordinate system and input it into the EfficientNet-B6 backbone network to extract multi-scale features, represented as follows: , (2) in This represents the pre-trained EfficientNet-B6 network. , {1, 2,3} , indicating the first k The output feature map of the layer, , and These represent the number of channels, height, and width of the feature map, respectively. Step 2-2-2: Flatten the feature map along the spatial dimension as follows: (3) This transformation ensures that each subsequent BEV grid point is mapped to its corresponding image projection coordinates. In this process, index queries are performed to achieve cross-domain feature sampling from the image domain to the BEV space; Steps 2-3, BEV space feature mapping based on projection index, specifically include the following steps: The three-dimensional coordinates of the center point of each voxel in the BEV mesh are represented as follows: . ; Step 2-3-1: Based on the camera's extrinsic and intrinsic parameters, convert the three-dimensional coordinates... Projecting onto the image plane yields its corresponding two-dimensional pixel coordinates, denoted as... ; Step 2-3-2, Calculation In the flattened feature map Linear indexes in: , (4) in The width of the feature map is given, and the indices of all valid projection points are combined into a single value. ; Step 2-3-3: Reconstruction of 3D Voxel Features Using index-based tensor sampling operations ( gather )exist The image features corresponding to each voxel are retrieved in the image and represented as follows: , (5) The sampled features were then re-sampled. reshape For the three-dimensional structure of BEV: , (5) in Z,X These are the length and width of the front area of ​​the vehicle, respectively. Y It is an estimated elevation; Step 3: BEV feature enhancement based on text semantic guidance: Step 3-1: Descriptive text generation based on a lightweight multimodal large language model: Step 3-1-1: Introduce a set of structured cue sets P to constrain the output content of the multimodal generative model; the cue set P is defined as a set of semantic cue templates, and its mathematical expression is: ， (6) Each prompt corresponds to a road semantic attribute description dimension, as shown in the following example: "Please describe the trend of longitudinal or transverse slope changes in the road surface." Please describe whether there are any potholes, cracks, or damaged areas, and explain their distribution. "Please describe whether there are areas of water accumulation or mirror-like reflections." "Please describe the visibility and integrity of lane lines, curbs, or other road structure markings." Step 3-1-2: Obtain the road surface image And the prompt set P as a common input to the multimodal generative model Generate corresponding descriptive text T Its generation process can be formally represented as: T= ( , P), (7) in , is a natural language sequence consisting of several words or sub-words, where L represents the length of the text sequence; Step 3-1-3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Input to the pre-trained text encoding model The preferred text encoding model is BERT or its improved version, used for context-sensitive deep semantic modeling of text sequences, i.e. , where serialized vector represents Each vector Indicates the first in the text i The semantic embedding of a word in a global context. For text feature dimensions; Step 3-2, BEV feature dimension transformation: The four-dimensional BEV features obtained in steps 2-3 are flattened and rearranged along the spatial and elevation dimensions; the Z, X, and Y dimensions are merged into a single sequence dimension, thereby converting the BEV features into a two-dimensional feature sequence representation. ,in , ; Step 3-3: Cross-attention calculation: Multi-scale BEV features As a query vector, Query uses textual semantic features As a key vector (Key) and a value vector (Value), it is represented as follows: , .. (8) in , and These are the learnable linear mapping matrices, d This is the scaling factor in the attention mechanism. This represents the BEV space feature representation enhanced by textual semantic features; Step 3-3-1, Nonlinear Mapping and Residual Connection: Output BEV features Apply feedforward network and residual connection: , (9) in The projection matrix used for dimension matching. The representation layer is normalized, and then the features are reshaped into a BEV 3D structure. Step 4, Design of the pseudo-binocular fusion module: Since real-world road reconstruction scenarios typically only provide monocular images or sparse depth information, directly constructing traditional binocular disparity volumes has objective limitations. This invention proposes a pseudo-auxiliary view generation method based on feature space translation and geometric consistency constraints. By constructing multiple "hypothetical viewpoints" within the BEV feature domain, it simulates feature correspondences under different disparity assumptions. Step 4-1, Main View Feature Definition: Define the multimodal augmentation feature set output in Step 3. Perform a copy operation and define it as a main view feature, i.e. . ; Step 4-2, Pseudo-auxiliary feature generation: Step 4-2-1, Spatial Saliency Perception Unit: First, Channel adaptation is performed using 1×1 convolution, followed by extraction of local geometric continuity in the BEV mesh using 3×3 convolution: , (10) Then, a saliency-enhanced output is generated using a standard spatial attention mechanism, as follows: , (11) in, They represent in The output responses of global average pooling and global max pooling operations performed on the feature map are concatenated along the channel dimension and then further reduced in dimensionality using a 3×3 convolution operation. This is then processed by... The sigmoid activation function is used to generate a saliency weight map, and the results are compared with... Performing a dot product operation yields the final spatial saliency-enhancing features. ; Step 4-2-2, Decoupling the intrinsic direction of lightweighting: First, in order to map the features to an orientation-sensitive, separable intermediate subspace, a channel-compressed convolution is performed, represented as: ( )), (8) in Indicates the normalization layer; Then, decoupling is performed in the horizontal and vertical directions respectively: , (9) , (10) in, Convolutional layer representation only uses The strip-shaped core slides in the height direction. Then it means to use The transverse strip core slides in the width direction; finally, using 3 Convolution fuses long-range responses and local compensations within a two-dimensional local domain, generating directional and spatially consistent feature maps. and ; Step 4-2-3, Structural splicing and one-dimensional convolutional recombination: The above three types of features are concatenated according to the channel dimension: . (11) Enhance features Reorganize the BEV network location into a serialized representation; first, flatten it in the spatial dimension: , (16) in The number of channels output by the hybrid convolutional stream is given; then, one-dimensional convolution is used to perform context aggregation and structural reorganization on the serialized features: , (17) in This represents a one-dimensional convolution operator performed along the sequence dimension; finally, it is reshaped back to the BEV space structure, and its output is defined as a pseudo-auxiliary view feature. ; Step 4-3: Construction of the learnable cost body: Obtaining main view features and pseudo-auxiliary features Subsequently, a learnable cost volume construction operator for monocular BEV scenarios is proposed. This operator explicitly constructs a pseudo-feature cost volume with geometric discrimination capability by introducing feature difference modeling, nonlinear gated attention, and an adaptive offset mechanism inspired by stereo parallax, without real second-view input. Step 5: Road elevation prediction and 3D reconstruction output: After completing the construction of the pseudo-feature cost volume, based on a reliable elevation regression head, the feature representation in the BEV space is used to predict the elevation at each location, and the three-dimensional geometry of the road is recovered accordingly.

2. The road reconstruction method based on a direction-aware pseudo-binocular network according to claim 1, characterized in that... Steps 1-2 specifically include the following steps: Step 1-2-1: Clip the spatial range of the 32-beam lidar point cloud data along the X and Y axes of the vehicle coordinate system. The clipping intervals are set to [-1.0m, 0.9m] and [2.1m, 7.1m], respectively, retaining the data of the effective road surface area in front of the road. Step 1-2-2: Divide the cropped irregular point cloud into regular grid units with a size of 3cm×3cm; Steps 1-2-3: Use the average elevation value of all points in the grid as the GT elevation corresponding to the grid, and set a binary mask to record the missing areas of no point data to obtain a regularized road surface elevation ground value raster.

3. The road reconstruction method based on a direction-aware pseudo-binocular network according to claim 1, characterized in that... Step 4-3 specifically includes the following steps: Step 4-3-1, Feature difference modeling of main and secondary views: Element-wise difference operations are performed on the main view features and pseudo-auxiliary features within the BEV space to explicitly characterize their inconsistencies in spatial structure and geometric distribution. That is: , (18) Step 4-3-2, Mask generation based on nonlinear gating: First, for difference features Perform a global max pooling operation to compress its spatial dimensionality and extract the most salient response information to obtain a global description vector. Subsequently, the pooled feature vectors are input into a nonlinear mapping module consisting of two multilayer perceptrons to achieve feature selection and gating modeling. This process is formally expressed as: , (19) in, , This represents the learnable weight matrix. ReLU represents the nonlinear activation function of the intermediate layer. express sigmoid Activation function, output A normalized dual-channel weight mask, used for subsequent feature weighting and offset modeling; Step 4-3-3, Adaptive Shift and Feature Weighting: mask m The two channels correspond to the weighting coefficients of the main view features and the pseudo-auxiliary features, respectively: , (20) Then, the features are reweighted through element-wise multiplication to obtain the enhanced feature representation: = , = , (21)； Step 4-3-4, Pseudo-binocular feature fusion: The weighted main view features and pseudo-auxiliary features are concatenated along the channel dimension to form a local geometric encoding representation: , (22) Should It is defined as the pseudo-feature cost volume in a monocular BEV scenario.

4. The road reconstruction method based on a direction-aware pseudo-binocular network according to claim 1, characterized in that... Step 5 specifically includes the following steps: Step 5-1, Feature encoding of the elevation regression head: To model the local-global elevation correlation within the BEV space, an elevation regression network commonly used in road reconstruction methods is introduced to perform layer-by-layer feature transformation on the cost volume. , (23) in, This represents a context encoding function composed of multiple convolutional layers and normalized units. This is a mid-level elevation-sensing feature representation used to regress the elevation value for each BEV grid; Step 5-2, Road Elevation Continuous Regression Prediction: By introducing a lightweight elevation regression head Continuous elevation prediction is performed for each BEV grid location, and this is done using the binary mask constructed in step 2. Constraints are applied to the output to ensure that the prediction results are only effective within road areas where real-world supervision exists, thus obtaining valid predicted road elevation results. It is expressed as follows: , (24) Step 5-3, Output of 3D road surface reconstruction: Based on the definition of the BEV mesh's position in the spatial coordinate system, the predicted elevation is mapped back to three-dimensional space to obtain a three-dimensional point set representation of the road surface: in, The coordinates of the BEV mesh in the horizontal plane; The final output is the 3D road reconstruction result; Step 5-4, Loss Function: During the training phase, to constrain the consistency between predicted and actual elevations, the supervised elevation regression loss, widely used in existing road elevation regression tasks, is employed: in Marking the actual road elevation. Indicates the number of valid grid cells.

Images