Self-supervised multi-modal fusion and collaborative optimization method suitable for curved and ramp scenarios

Abstract

The application relates to a self-supervised multi-modal fusion and collaborative optimization method suitable for curved and ramp scenes, comprising the following steps: collecting original perception images; generating four-way recovery images with geometric consistency; constructing a multi-modal fusion model to generate a depth confidence map aligned with the recovery images; obtaining unified ground plane parameters; collecting all coordinate points to construct an initial 3D lane candidate point set; generating a high-density locally enhanced BEV lane representation; outputting a fused global consistency 3D lane map; and outputting structured 3D lane line labeling data. The method can accurately depict non-planar structures such as curved superhighways and longitudinal ramps, greatly improving the geometric consistency in complex road scenes; effectively alleviates the problems of single vehicle occlusion, sensor noise and the like, significantly improves the accuracy, integrity and system-level reliability of 3D lane reconstruction, and avoids the dependence on high-precision maps or 3D true value labeling, and has strong engineering landing value.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving perception technology, and in particular to a self-supervised multimodal fusion and collaborative optimization method applicable to curve and slope scenarios. Background Technology

[0002] A key technology in autonomous driving and advanced driver assistance systems (ADAS) is accurate 3D lane line perception. However, most current methods for 3D lane line detection rely on large-scale real-world 3D labeled datasets or geometric models based on idealized road assumptions. While methods based on large-scale real-world 3D labeled datasets offer high accuracy, their data acquisition and annotation costs are extremely high. Furthermore, methods based on geometric models often encounter the following problems in practical applications:

[0003] First, existing 3D lane detection models rely heavily on training data from normal scenarios such as sunny days. Their perception capabilities are significantly reduced for images under adverse weather conditions such as rain, snow, fog, and night, resulting in insufficient reliability of the system in extreme environments.

[0004] Second, mainstream methods generally adopt the simplified assumption that "the ground is an ideal plane". When vehicles travel on non-planar roads with longitudinal slopes, lateral curves with superelevation, or undulating terrain, their 3D reconstruction results will produce serious geometric distortions and cannot accurately reflect the real road shape.

[0005] Third, while existing image restoration methods can improve image quality under adverse weather conditions such as rain, snow, and fog, their output is only 2D visual enhancement results and cannot provide geometrically accurate 3D lane markings. At the same time, these restoration models are usually designed independently of the 3D perception module and are not optimized in coordination with geometric perception sources such as radar and V2X, making it difficult to achieve high-precision and self-consistent 3D lane reconstruction in non-planar road scenarios.

[0006] Therefore, there is an urgent need to develop a self-supervised multimodal fusion and collaborative optimization method for curves and slopes under all weather conditions. Summary of the Invention

[0007] To address the problems of severe degradation in visual perception performance under adverse weather conditions such as rain, snow, and fog, difficulty in accurately modeling non-planar road geometry such as curves, superelevations, and longitudinal slopes, reliance on expensive 3D annotation or high-precision maps, and insufficient robustness due to occlusion and noise affecting single-vehicle perception, this invention aims to provide a self-supervised multimodal fusion and collaborative optimization method applicable to curve and slope scenarios. This method effectively suppresses the degradation effect of adverse weather conditions such as rain, snow, and fog on visual input, significantly improves the quality of front-end perception, and significantly enhances the accuracy, completeness, and system-level reliability of 3D lane reconstruction.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a self-supervised multimodal fusion and collaborative optimization method applicable to curve and ramp scenarios, the method comprising the following sequential steps:

[0009] (1) Acquiring raw sensory images Simultaneously collect radar point cloud data;

[0010] (2) The original perceptual image An image containing lane line information and affected by adverse weather conditions such as rain, snow, and fog is input into a lightweight image restoration network, which generates a geometrically consistent four-way restored image. ;

[0011] (3) Construct a multimodal fusion model to restore the four-channel images The corresponding radar point cloud data is input into a multimodal fusion model to obtain a multi-scale feature map F, which generates a depth confidence map aligned with the reconstructed image. ;

[0012] (4) Based on the multi-scale feature map F and the depth confidence map Obtain unified ground plane parameters;

[0013] (5) Construct an initial 3D lane candidate point set based on unified ground plane parameters;

[0014] (6) Map each point in the initial 3D lane candidate point set to the BEV space to generate a high-density locally enhanced BEV lane representation. ;

[0015] (7) Construct a V2X cooperative perception system, model the initial 3D lane candidate point set of each vehicle into a structured graph with confidence, and output the fused globally consistent 3D lane map;

[0016] (8) Interpolate and geometrically optimize the globally consistent 3D lane map to finally output structured 3D lane line annotation data.

[0017] Step (1) specifically refers to: acquiring raw sensory images through four cameras installed in the vehicle. The four cameras respectively cover the front, left, right, and rear views of the vehicle, recording the... The road image is the original perceived image. , H represents the original perceived image. The height, W, represents the original perceived image. The width corresponds to the camera intrinsic parameter matrix. The extrinsic parameter matrix is ,in, For the first Rotation matrix of road cameras; For the first The translation vector of the road camera. Represents the set of rigid body transformations in three-dimensional space; radar point cloud data is acquired simultaneously via vehicle-mounted millimeter-wave radar. , This represents the three-dimensional coordinates of the j-th millimeter-wave radar point in the vehicle coordinate system; This represents the total number of points in a single frame of millimeter-wave radar point cloud; all points are represented in the vehicle coordinate system; raw perception images collected by four cameras. Strict synchronization with radar point cloud data is achieved through hardware triggering or timestamp alignment. The synchronization is achieved through CAN bus timestamp alignment, with a time error of less than 50ms.

[0018] Step (2) specifically includes the following steps in sequence:

[0019] (2a) The lightweight image restoration network adopts an encoder-decoder architecture, which includes a shallow encoder with shared weights, a lightweight feature fusion module, a Transformer encoder, an asymmetric Transformer decoder, an upsampling projection head, and a geometric consistency constraint output layer.

[0020] The shared-weight shallow encoder consists of three convolutional blocks, each containing a 3×3 convolutional layer, batch normalization, and a ReLU activation function, used to extract data from various raw perceptual images. Extracting multi-scale low-dimensional features;

[0021] The lightweight feature fusion module adaptively enhances the feature response of image regions severely degraded by rain, snow, and fog by introducing a channel attention mechanism.

[0022] The Transformer encoder divides the input into non-overlapping image blocks and embeds them into a token sequence. Deep features are extracted through L-level backbone blocks, and an in-chip Transformer block is inserted after each level.

[0023] The asymmetric Transformer decoder uses three learnable weather type query embeddings. As input, contextual information is extracted from the output of the shallow encoder through cross-attention:

[0024] ;

[0025] in, For decoder output; For cross-attention functions; Represents a weather query matrix; This represents the key matrix output by the shallow encoder. , This is the output of a shallow encoder. This represents the projection matrix that projects the shallow encoder features onto the key space; This represents the value matrix output by the shallow encoder. , This represents the projection matrix that projects the features of the shallow encoder onto the value space;

[0026] The upsampling projection head fuses the decoder output with features from each stage of the shallow encoder through skip connections, and then gradually upsamples them to the original resolution via transposed convolution to generate four-channel restored images. ;

[0027] Geometric consistency constraints on the output layer for reconstructing the image Introducing cross-viewpoint photometric consistency loss, for any pair of adjacent cameras with overlapping fields of view, the known camera extrinsic parameters are used to... Road restoration image projection to the first The image plane is used to generate a composite image and compare it with the first image plane. Actual road reconstruction image Pixel-level comparison is performed in overlapping areas to constrain the restoration process from distorting the multi-view geometry;

[0028] (2b) Original perceived image First, a true lane line image is obtained through color thresholding, mask generation, and bitwise AND operation; then, edge images are obtained through Canny edge detection. Finally, a combined image integrating lane line semantics and geometric edge information is generated through bitwise OR operations. :

[0029] ;

[0030] in, Indicates channel multiplication; Indicates bitwise OR; Represents the semantic mask for lane lines;

[0031] (2c) The original perceptual image As the main input, after normalization, it is divided into... There are three non-overlapping image patches, each of size P*P, where H represents the original perceptual image. The height, W, represents the original perceived image. The width of the image patch is P, where P represents the side length of the image patch. Each image patch is flattened and subjected to a learnable linear projection to obtain the initial token sequence. :

[0032] ;

[0033] in, For embedding matrix, For position encoding, For sequence length, For embedded dimensions, For the first The original representation of the input token sequence;

[0034] (2d) will The input is a Transformer encoder with L stacked layers; the output of the Lth stage is:

[0035] ;

[0036] ;

[0037] Among them, MSA stands for Multi-Head Self-Attention, LN stands for Layer Normalization, and MLP stands for Two-Layer Fully Connected Network. Indicates the first -1 layer output; This represents the intermediate output after self-attention; Indicates the first Layer output;

[0038] After each stage output, an in-chip multi-head self-attention is inserted to obtain a feature representation that enhances the spatial context:

[0039] ;

[0040] in, Represents the query matrix. Represents the key matrix, Represents a value matrix, This represents the dot product of the query and the key. Indicates the scaling factor; -MSA indicates multi-head self-attention within the film.

[0041] Step (3) specifically includes the following steps in sequence:

[0042] (3a) Construct a multimodal fusion model, which includes a ResNet visual encoder, a radar depth confidence map generation module, and a cross-modal adaptive fusion module. The ResNet visual encoder is composed of a lightweight convolutional neural network with shared weights. The radar depth confidence map generation module includes a point cloud preprocessing unit and a projection rendering unit. The cross-modal adaptive fusion module includes a channel-space gating unit, which includes a 1×1 convolutional layer and a weighted modulation unit. The weighted modulation unit includes a channel weight generation branch, a spatial weight generation branch, and a dual modulation module.

[0043] (3b) For each restored image, multi-scale visual features are extracted using the ResNet visual encoder in the multimodal fusion model to obtain a multi-scale feature map F: For each restored image Multi-scale feature maps F are extracted using a ResNet visual encoder. , and These represent the restored images. Height and width;

[0044] (3c) Simultaneously, the radar point cloud data is projected onto the image plane corresponding to each camera to generate a depth confidence map aligned with the restored image. Radar point cloud data constitutes a radar point cloud data set. ,Will Project to the Generate depth confidence maps from image planes. , This indicates the height of the depth confidence map. This represents the width of the depth confidence map; where, Indicates the total number of radar point clouds; Let be the three-dimensional coordinates of the j-th millimeter-wave radar point in the vehicle coordinate system.

[0045] Step (4) specifically includes the following steps in sequence:

[0046] (4a) Depth confidence map Channel concatenation with the multi-scale feature map F yields a fused feature map. :

[0047] ;

[0048] In the formula, This indicates a splicing operation at the channel level; Indicates from the first Road restoration image Multi-scale feature maps extracted from them; Indicates the first Depth confidence map from the perspective of a road camera; C represents the number of visual channels; Indicates the height of the fused feature map; Indicates the width of the fused feature map;

[0049] (4b) Fusion feature map Global average pooling is performed, and the horizon normal vector is predicted using a lightweight regression head MLP. and offset , where the horizon normal vector From longitudinal slope angle and lateral super high angle Parameterization yields:

[0050] ;

[0051] ;

[0052] ;

[0053] in, Indicates a local ground plane; ; For any three-dimensional point in the world coordinate system; Indicates the offset; The input feature vector is represented by MLP, which stands for Multilayer Perceptron. Indicates the longitudinal tilt component. Indicates the principal component in the vertical direction. Indicates the lateral tilt component;

[0054] (4c) Calculate the attention weights of the four-way estimation Weighted fusion yields unified ground plane parameters:

[0055] ;

[0056] ;

[0057] In the formula, Indicates the longitudinal slope angle. Indicates the lateral elevation angle. Indicates the first The longitudinal slope angle estimated by each camera. Indicates the first The estimated lateral elevation angle from each camera. Indicates the first The estimated weights; and Consolidate unified ground plane parameters; The representation of the first The characteristics of road cameras, This represents a learnable query vector. Indicates the calculation of the first The dot product of path features and global query.

[0058] Step (5) specifically includes the following steps in sequence:

[0059] (5a) From the combined image Extract salient lane point set ,Pick The center of the connected domain; where, Indicates the first Street camera The pixel coordinates of a significant lane point Horizontal pixel coordinates; Vertical pixel coordinates; Indicates the first The number of salient lane points extracted from the road image. A salient lane point is a point that simultaneously satisfies the following conditions: it is located within the semantic region of the lane line, has a strong local geometric edge response, and is selected as a representative center point of the local structure in the connected component analysis. This represents the semantic mask for lane lines. Represents the edge image;

[0060] (5b) For each point Construct a normalized line of sight :

[0061] ;

[0062] in, For the first The intrinsic parameter matrix of the road camera;

[0063] Calculating the scale factor of the intersection point of a ray and a plane using geometric back projection :

[0064] ;

[0065] in, It is the normal vector of the ground plane; Indicates the offset; For the first Rotation matrix of road cameras; For the first The translation vector of the road camera;

[0066] Solving for 3D world coordinates:

[0067] ;

[0068] in, Indicates the first The first observation by the road camera 3D world coordinates of a point;

[0069] Repeat steps (5a) and (5b) for all 2D lane points to obtain the set of 3D world coordinates, which is the initial 3D lane candidate point set. .

[0070] Step (6) specifically includes the following steps in sequence:

[0071] (6a) Construct a slope-aware adaptive resolution BEV map based on terrain slope information, and define the physical spatial range of the BEV map as follows: rice, rice, The horizontal direction of the vehicle coordinate system. This represents the longitudinal direction of the vehicle coordinate system; Represents the maximum horizontal distance. =15; Represents the maximum vertical distance. =50;

[0072] (6b) Setting the base grid resolution for distance perception: Along the longitudinal Y-axis, the base grid resolution decreases as distance increases; specifically, the longitudinal range... Divide into U intervals, with the vertical resolution of the u-th interval being... ,u∈U, Let be the center distance of the u-th interval. and All are linear coefficients controlling the resolution decay rate; the horizontal X-axis uses a fixed base resolution. rice;

[0073] (6c) Fuse feature maps Using the camera extrinsic matrix and unified ground plane parameters, the image is projected onto the BEV space through a differentiable view transformation, and cross-view aggregation is provided to generate dense BEV feature maps. A spatial prior module is constructed, which includes a geometric deviation map generation module and a weighted prediction head. The geometric deviation map generation module calculates the signed distance from the world coordinates of the center point of each BEV cell to the ground plane, generating a geometric deviation map. This geometric deviation map is then compared with... The weight prediction head consists of a 3×3 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a sigmoid function connected in sequence, mapping the fused features to an attention heatmap; the input to the spatial prior module is a dense BEV feature map. Unify ground plane parameters and offset d, and output a BEV spatial weight map. ,in, This represents the height of the BEV spatial weight map, i.e., the vertical resolution. This represents the width of the BEV spatial weight map, i.e., its horizontal resolution.

[0074] (6d) Perform slope- or curvature-aware dynamic resolution enhancement on top of the base mesh: For each base mesh cell in the BEV space, double the local mesh resolution of each base mesh cell in the BEV space if any of the following three conditions are met, i.e. , ,in, Indicates adaptive horizontal resolution. Indicates adaptive vertical resolution. Indicates the basic horizontal resolution. This represents the basic vertical resolution; the three conditions include condition 1, condition 2, and condition 3.

[0075] Condition 1: Attention weights corresponding to basic grid cells ,in The preset threshold;

[0076] Condition 2: Absolute value of the longitudinal slope angle of the road ,in ;

[0077] Condition 3: Absolute value of the road's lateral superelevation angle ,in ;

[0078] in, Indicates the longitudinal slope sensitivity threshold. This indicates an extremely high sensitivity threshold in the horizontal direction;

[0079] (6e) Based on the adaptive resolution network at different distances, a diffusion feature allocation is performed on the initial 3D lane candidate point set to construct a high-fidelity BEV lane response map R;

[0080] (6f) Normalize and smooth the BEV response map R to obtain a high-density locally enhanced BEV lane representation. .

[0081] Step (7) specifically includes the following steps in sequence:

[0082] (7a) The V2X cooperative perception system includes a local perception unit, a V2X communication interface, and a cross-vehicle geometry alignment module. The local perception unit is configured in each participating vehicle and outputs locally estimated unified ground plane parameters and a locally reconstructed 3D lane candidate point set. All data are represented in the vehicle's own coordinate system. The V2X communication interface broadcasts the perception results of the vehicle to neighboring vehicles through DSRC dedicated short-range communication and C-V2X cellular vehicle networking standard protocol, and receives corresponding data from other vehicles to form a multi-view observation set. The input of the cross-vehicle geometry alignment module comes from the local results of M vehicles, and then constructs a ground plane consistency map, calculates the maximum common subgraph, and jointly optimizes the ground plane parameters.

[0083] (7b) In the V2X cooperative perception system, the 3D lane point set of each vehicle v is modeled as a structured graph. Node set It includes all detected 3D lane points and their confidence scores, while the edge set... This represents the topological relationship between 3D lane points that belong to the same lane and are spatially adjacent or have a continuous set relationship;

[0084] Node set For the first 3D lane points, For reliability confidence; edge set This indicates the topological connection between lane points. If two points belong to the same lane and have adjacent numbers, then there is an edge connection.

[0085] (7c) Reliability confidence Joint estimation of quality from multi-source sensing:

[0086] ;

[0087] in, Indicates image restoration The input is the classification score output by a lightweight lane confidence estimation module, which uses the ultrafast structure-aware deep lane detection model UFLD. This represents the radar depth confidence level for the corresponding image location. This is a 3D to 2D projection function. For weighting;

[0088] (7d) For any two cars and Perform uncertainty-aware maximum common subgraph matching to maximize the geometric and topological consistency of the matched subgraphs;

[0089] (7e) Define node pairs Weighted geometric similarity for:

[0090] ;

[0091] in, For vehicles arrive The relative pose transformation matrix is ​​obtained by solving the global navigation satellite system shared by V2X; The geometric tolerance scale; The vehicle The Reliability confidence level of each lane node; Indicates vehicle The Reliability confidence level of each lane node; This represents the natural exponential function. Indicates vehicle The Middle The coordinates of each lane point Indicates vehicle The Middle The coordinates of each lane point;

[0092] (7f) Model the matching problem as a combinatorial optimization with topological constraints:

[0093] ;

[0094] ;

[0095] ;

[0096] in, Represents a binary matching variable. Indicates vehicle The number of candidate lane points, Indicates vehicle The number of candidate lane points;

[0097] If the vehicle Middle node and They are adjacent, that is ( , () is a vehicle Lane map One of the edges in, and ( , Vehicles were matched separately. nodes and ,So and It must also be in the vehicle In the lane map, adjacent lanes must be matched to ensure that the adjacency relationship between lane points is consistent.

[0098] (7g) The alignment problem of lane point sets between vehicles is modeled as a maximum common subgraph optimization problem. Its objective is to maximize the weighted similarity of matching node pairs while maintaining the consistency of the graph topology. The maximum common subgraph optimization problem can be formalized as a maximum weighted subgraph isomorphism problem defined on a bipartite graph with higher-order structure constraints, where the two parts of the bipartite graph are the vehicles and the lane points. and The lane nodes are identified; the Hungarian algorithm is used to solve for the optimal matching pair set. ;

[0099] (7h) Based on the set of optimal matching pairs Align and merge the 3D lane points of multiple vehicles to obtain a globally consistent 3D lane map.

[0100] Step (8) specifically includes the following steps in sequence:

[0101] (8a) In the BEV space, interpolation and geometric optimization are performed on the globally consistent 3D lane map. A complete end-to-end network model, including a lightweight image restoration network, a multimodal fusion model, and a V2X cooperative perception system, is trained using a total loss function that relies solely on 2D supervision. Relying solely on 2D lane line annotations, the expression is:

[0102] ;

[0103] in, Loss due to 2D lane detection monitoring; , These are all preset hyperparameters used to balance the weights of various losses; For 3D curvature smoothing constraint loss; This represents the lane topology consistency loss.

[0104] (8b) 3D curvature smoothing constraint loss The calculation formula is:

[0105] ;

[0106] Where K is the total number of lanes predicted in the current frame; For the first The number of 3D points on each lane; For the first The first lane The coordinates of a 3D point; Represents the L2 norm;

[0107] (8c) Lane topology consistency loss The calculation formula is:

[0108] ;

[0109] Where y is a predefined set of vertical sampling positions; This represents the number of sampling locations; For the first The lateral coordinates of each lane at the longitudinal distance Y=y are obtained through interpolation; Var( ) indicates variance operation;

[0110] (8d) Introducing a new alignment loss To ensure the high-fidelity BEV lane response map R matches the final output adaptive resolution BEV lane feature map. The consistency is calculated using the following formula:

[0111] ;

[0112] in, This indicates the vertical resolution of the BEV mesh. This indicates the lateral resolution of the BEV grid. Indicates the vertical grid row index, Indicates the horizontal grid column index. Indicates the lane response value of the BEV. This indicates a BEV lane with high-density, locally enhanced designation.

[0113] (8e) will Add to the total loss function to obtain the updated total loss function. :

[0114] ;

[0115] in, It is a new preset hyperparameter used to control the degree of influence on its loss;

[0116] (8f) Using the updated total loss function The complete end-to-end network model is jointly optimized to output structured 3D lane line annotation data, thereby improving the accuracy and geometric robustness of 3D lane line reconstruction on non-planar roads using only 2D lane annotations.

[0117] As can be seen from the above technical solution, the beneficial effects of this invention are as follows: First, by introducing a lightweight image restoration network, the degradation effect of adverse weather conditions such as rain, snow, and fog on visual input is effectively suppressed, significantly improving the front-end perception quality and providing a clear and reliable image foundation for subsequent lane detection, thus solving the problem of the sharp performance decline of traditional methods in low visibility scenarios. Second, an explicit ground plane parameter estimation mechanism is proposed, which can accurately characterize non-planar structures such as curve superelevation and longitudinal slopes, overcoming the 3D reconstruction distortion caused by the assumption that the road is a horizontal plane in existing methods, and greatly improving the geometric consistency in complex road scenarios. Third, a self-supervised multimodal fusion and V2X collaborative optimization framework is designed. Under the premise of relying only on 2D lane annotation, the feature robustness is enhanced by radar-visual complementary fusion, and cross-vehicle geometric alignment is performed using ground plane parameters shared by multiple vehicles and 3D lane maps, effectively alleviating problems such as single-vehicle occlusion and sensor noise, significantly improving the accuracy, completeness, and system-level reliability of 3D lane reconstruction, while avoiding dependence on high-precision maps or 3D ground truth annotations, and possessing strong engineering application value. Attached Figure Description

[0118] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0119] like Figure 1 As shown, a self-supervised multimodal fusion and collaborative optimization method applicable to curve and ramp scenarios is presented. The method includes the following sequential steps:

[0120] (1) Acquire raw sensor images through four cameras installed in the vehicle. The vehicle-mounted millimeter-wave radar synchronously collects radar point cloud data;

[0121] (2) The original perceptual image An image containing lane line information and affected by adverse weather conditions such as rain, snow, and fog is input into a lightweight image restoration network, which generates a geometrically consistent four-way restored image. ;

[0122] (3) Construct a multimodal fusion model to restore the four-channel images The radar point cloud data is input into the multimodal fusion model; for each recovered image, multi-scale visual features are extracted using the ResNet visual encoder in the multimodal fusion model to obtain a multi-scale feature map F; simultaneously, the radar point cloud data is projected onto the image plane corresponding to each camera to generate a depth confidence map aligned with the recovered image. ;

[0123] (4) Combine the multi-scale feature maps F of each path with the depth confidence map The fused feature map is obtained by stitching along the channel. After global average pooling, the data is input into a lightweight regression head MLP to predict local ground plane parameters. The four estimation results are then weighted by attention weights to obtain unified ground plane parameters.

[0124] (5) For any 2D lane point, use the camera intrinsic matrix, extrinsic matrix and unified ground plane parameters to calculate the 3D world coordinates through geometric back projection, and collect all coordinate points to construct an initial 3D lane candidate point set;

[0125] (6) Construct a slope-aware adaptive resolution BEV map based on terrain slope information, dynamically allocate higher grid density in key areas, map each point in the initial 3D lane candidate point set to the BEV space, and propagate the geometric and semantic information of each point in the initial 3D lane candidate point set to the k nearest neighbor BEV grid cells to generate a high-density locally enhanced BEV lane representation. ;

[0126] (7) Construct a V2X collaborative perception system, model the initial 3D lane candidate point set of each vehicle into a structured graph with confidence, and use the maximum common subgraph matching with uncertainty perception to perform cross-vehicle alignment. The matching process uses node confidence weighted geometric similarity to automatically suppress low reliability observations caused by bad weather or occlusion, and outputs a fused global consistent 3D lane map.

[0127] (8) In the BEV space, interpolation and geometric optimization are performed on the globally consistent 3D lane map. The total loss function that relies only on 2D supervision is used to train a complete end-to-end network model including a lightweight image restoration network, a multimodal fusion model and a V2X cooperative perception system. Finally, structured 3D lane line annotation data is output.

[0128] Step (1) specifically refers to: the four cameras respectively cover the front, left, right and rear views of the vehicle, and record the first... The road image is the original perceived image. , H represents the original perceived image. The height, W, represents the original perceived image. The width corresponds to the camera intrinsic parameter matrix. The extrinsic parameter matrix is ,in, For the first Rotation matrix of road cameras; For the first The translation vector of the road camera. Represents the set of rigid body transformations in three-dimensional space; the synchronously acquired radar point cloud data is , This represents the three-dimensional coordinates of the j-th millimeter-wave radar point in the vehicle coordinate system; This represents the total number of points in a single frame of millimeter-wave radar point cloud; all points are represented in the vehicle coordinate system; raw perception images collected by four cameras. Strict synchronization with radar point cloud data is achieved through hardware triggering or timestamp alignment. This synchronization is accomplished via CAN bus timestamp alignment, with a time error of less than 50ms. The original perceived image... The system uses four wide-angle cameras installed at the front, left, right, and rear to simultaneously capture single-frame RGB images, covering a 360° horizontal field of view around the vehicle.

[0129] Step (2) specifically includes the following steps in sequence:

[0130] (2a) The lightweight image restoration network adopts an encoder-decoder architecture, which includes a shallow encoder with shared weights, a lightweight feature fusion module, a Transformer encoder, an asymmetric Transformer decoder, an upsampling projection head, and a geometric consistency constraint output layer.

[0131] The shared-weight shallow encoder consists of three convolutional blocks, each containing a 3×3 convolutional layer, batch normalization, and a ReLU activation function, used to extract data from various raw perceptual images. Extracting multi-scale low-dimensional features;

[0132] The lightweight feature fusion module adaptively enhances the feature response of image regions severely degraded by rain, snow, and fog by introducing a channel attention mechanism.

[0133] The Transformer encoder divides the input into non-overlapping image blocks and embeds them into a token sequence. Deep features are extracted through L-level backbone blocks, and an in-chip Transformer block is inserted after each level.

[0134] The asymmetric Transformer decoder uses three learnable weather type query embeddings. As input, contextual information is extracted from the output of the shallow encoder through cross-attention:

[0135] ;

[0136] in, For decoder output; For cross-attention functions; Represents a weather query matrix; This represents the key matrix output by the shallow encoder. , This is the output of a shallow encoder. This represents the projection matrix that projects the shallow encoder features onto the key space; This represents the value matrix output by the shallow encoder. , This represents the projection matrix that projects the features of the shallow encoder onto the value space;

[0137] This enables a single model to adaptively respond to any input weather without the need for preset labels or switching sub-networks.

[0138] The upsampling projection head fuses the decoder output with features from each stage of the shallow encoder through skip connections, and then gradually upsamples them to the original resolution via transposed convolution to generate four-channel restored images. ; Restore image Without relying on any paired real-world clear-day reference images, data can be obtained through self-supervised training on datasets containing only severe weather images, significantly reducing data acquisition and annotation costs.

[0139] Geometric consistency constraints on the output layer for reconstructing the image Introducing cross-viewpoint photometric consistency loss, for any pair of adjacent cameras with overlapping fields of view, the i-th restored image is projected onto the i-th path using known camera extrinsic parameters. The image plane is used to generate a composite image and compare it with the first image plane. Actual road reconstruction image Pixel-level comparison is performed in overlapping areas to constrain the restoration process from distorting the multi-view geometry;

[0140] (2b) Original perceived image First, a true lane line image is obtained through color thresholding, mask generation, and bitwise AND operation; then, edge images are obtained through Canny edge detection. Finally, a combined image integrating lane line semantics and geometric edge information is generated through bitwise OR operations. :

[0141] ;

[0142] in, Indicates channel factorial; Indicates bitwise OR; Represents the semantic mask for lane lines;

[0143] (2c) The original perceptual image As the main input, after normalization, it is divided into... There are three non-overlapping image patches, each of size P*P, where H represents the original perceptual image. The height, W, represents the original perceived image. The width of the image patch is P, where P represents the side length of the image patch. Each image patch is flattened and subjected to a learnable linear projection to obtain the initial token sequence. :

[0144] ;

[0145] in, For embedding matrix, For position encoding, For sequence length, For embedded dimensions, For the first The original representation of each input token;

[0146] (2d) will The input is a Transformer encoder with L stacked layers; the output of the Lth stage is:

[0147] ;

[0148] ;

[0149] Among them, MSA stands for Multi-Head Self-Attention, LN stands for Layer Normalization, and MLP stands for Two-Layer Fully Connected Network. Indicates the first Layer output; This represents the intermediate output after self-attention; Indicates the first Layer output;

[0150] After each stage output, an in-chip multi-head self-attention is inserted to obtain a feature representation that enhances the spatial context:

[0151] ;

[0152] in, Represents the query matrix. Represents the key matrix, Represents a value matrix, This represents the dot product of the query and the key. Indicates the scaling factor; -MSA indicates multi-head self-attention within the film.

[0153] Step (3) specifically includes the following steps in sequence:

[0154] (3a) Construct a multimodal fusion model, which includes a ResNet visual encoder, a radar depth confidence map generation module, and a cross-modal adaptive fusion module. The ResNet visual encoder is composed of a lightweight convolutional neural network with shared weights. The radar depth confidence map generation module includes a point cloud preprocessing unit and a projection rendering unit. The cross-modal adaptive fusion module includes a channel-space gating unit, which includes a 1×1 convolutional layer and a weighted modulation unit. The weighted modulation unit includes a channel weight generation branch, a spatial weight generation branch, and a dual modulation module.

[0155] (3b) For each recovered image Multi-scale feature maps F are extracted using a ResNet visual encoder. , and These represent the restored images. Height and width;

[0156] (3c) Radar point cloud data constitutes a radar point cloud data set ,Will Project to the Generate depth confidence maps from image planes. , This indicates the height of the depth confidence map. This represents the width of the depth confidence map; where, Indicates the total number of radar point clouds; Let be the three-dimensional coordinates of the j-th millimeter-wave radar point in the vehicle coordinate system. Depth confidence map. Each pixel value represents the depth confidence provided by the radar at that image location.

[0157] The multimodal fusion model is designed specifically for curve and ramp scenarios under three adverse weather conditions. Its core objective is to fuse visually restored images with radar depth information to generate geometrically accurate and robust lane perception results.

[0158] Step (4) specifically includes the following steps in sequence:

[0159] (4a) Depth confidence map Channel concatenation with the multi-scale feature map F yields a fused feature map. :

[0160] ;

[0161] In the formula, This indicates a splicing operation at the channel level; Indicates from the first Road restoration image Multi-scale feature maps extracted from them; Indicates the first Depth confidence map from the perspective of a road camera; C represents the number of visual channels; Indicates the height of the fused feature map; Indicates the width of the fused feature map;

[0162] (4b) Fusion feature map Global average pooling is performed, and the horizon normal vector is predicted using a lightweight regression head MLP. and offset , where the horizon normal vector From longitudinal slope angle and lateral super high angle Parameterization yields:

[0163] ;

[0164] ;

[0165] ;

[0166] in, Indicates a local ground plane; ; For any three-dimensional point in the world coordinate system; Indicates the offset; The input feature vector is represented by MLP, which stands for Multilayer Perceptron. Indicates the longitudinal tilt component. Indicates the principal component in the vertical direction. Indicates the lateral tilt component;

[0167] (4c) Calculate the attention weights of the four-way estimation Weighted fusion yields unified ground plane parameters:

[0168] ;

[0169] ;

[0170] In the formula, Indicates the longitudinal slope angle. Indicates the lateral elevation angle. Indicates the first The longitudinal slope angle estimated by each camera. Indicates the first The estimated lateral elevation angle from each camera. Indicates the first The estimated weights; and Consolidate unified ground plane parameters; The representation of the first The characteristics of road cameras, This represents a learnable query vector. Indicates the calculation of the first The dot product of path features and global query.

[0171] Step (5) specifically includes the following steps in sequence:

[0172] (5a) From the combined image Extract salient lane point set ,Pick The center of the connected domain; where, Indicates the first Street camera The pixel coordinates of a significant lane point Horizontal pixel coordinates; Vertical pixel coordinates; Indicates the first The number of salient lane points extracted from the road image. A salient lane point is a point that simultaneously satisfies the following conditions: it is located within the semantic region of the lane line, has a strong local geometric edge response, and is selected as a representative center point of the local structure in the connected component analysis. This represents the semantic mask for lane lines. Represents the edge image;

[0173] (5b) For each point Construct a normalized line of sight :

[0174] ;

[0175] in, For the first The intrinsic parameter matrix of the road camera;

[0176] Calculating the scale factor of the intersection point of a ray and a plane using geometric back projection :

[0177] ;

[0178] in, It is the normal vector of the ground plane; Indicates the offset; For the first Rotation matrix of road cameras; For the first The translation vector of the road camera;

[0179] Solving for 3D world coordinates:

[0180] ;

[0181] in, Indicates the first The first observation by the road camera 3D world coordinates of a point;

[0182] Repeat steps (5a) and (5b) for all 2D lane points to obtain the set of 3D world coordinates, which is the initial 3D lane candidate point set. .

[0183] Step (6) specifically includes the following steps in sequence:

[0184] (6a) Define the physical space extent of the BEV diagram as follows: rice, rice, The horizontal direction of the vehicle coordinate system. This represents the longitudinal direction of the vehicle coordinate system; Represents the maximum horizontal distance. =15; Represents the maximum vertical distance. =50;

[0185] (6b) Setting the base grid resolution for distance perception: Along the longitudinal Y-axis, the base grid resolution decreases as distance increases; specifically, the longitudinal range... Divide into U intervals, with the vertical resolution of the u-th interval being... ,u∈U, Let be the center distance of the u-th interval. and All are linear coefficients controlling the resolution decay rate; the horizontal X-axis uses a fixed base resolution. rice;

[0186] (6c) Fuse feature maps Using the camera extrinsic matrix and unified ground plane parameters, the image is projected onto the BEV space through a differentiable view transformation, and cross-view aggregation is provided to generate dense BEV feature maps. A spatial prior module is constructed, which includes a geometric deviation map generation module and a weighted prediction head. The geometric deviation map generation module calculates the signed distance from the world coordinates of the center point of each BEV cell to the ground plane, generating a geometric deviation map. This geometric deviation map is then compared with... The weight prediction head consists of a 3×3 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a sigmoid function connected in sequence, mapping the fused features to an attention heatmap; the input to the spatial prior module is a dense BEV feature map. Unify ground plane parameters and offset d, and output a BEV spatial weight map. ,in, This represents the height of the BEV spatial weight map, i.e., the vertical resolution. This represents the width of the BEV spatial weight map, i.e., its horizontal resolution.

[0187] (6d) Perform slope- or curvature-aware dynamic resolution enhancement on top of the base mesh: For each base mesh cell in the BEV space, double the local mesh resolution of each base mesh cell in the BEV space if any of the following three conditions are met, i.e. , ,in, Indicates adaptive horizontal resolution. Indicates adaptive vertical resolution. Indicates the basic horizontal resolution. This represents the basic vertical resolution; the three conditions include condition 1, condition 2, and condition 3.

[0188] Condition 1: Attention weights corresponding to basic grid cells ,in The preset threshold;

[0189] Condition 2: Absolute value of the longitudinal slope angle of the road ,in ;

[0190] Condition 3: Absolute value of the road's lateral superelevation angle ,in ;

[0191] in, Indicates the longitudinal slope sensitivity threshold. This indicates an extremely high sensitivity threshold in the horizontal direction;

[0192] (6e) Based on an adaptive resolution network at different distances, a diffusion-based feature allocation is performed on the initial 3D lane candidate point set to construct a high-fidelity BEV lane response map, specifically:

[0193] (6e1) For the first The first observation by the road camera 3D world coordinates of a point Projected onto the BEV plane ; Indicates vertical position. Indicates altitude, Indicates horizontal position. Indicates the first The first observation by the road camera The two-dimensional projected coordinates of a point on the BEV plane;

[0194] (6e2) Find in the BEV mesh The former The nearest neighbor grid centers form a set. ; Indicates the first The nearest neighbor grid center;

[0195] (6e3) Calculate the allocation weights :

[0196] ;

[0197] in, , For diffusion variance, Determined by both local road geometry and point confidence:

[0198] ;

[0199] in, , , All are preset hyperparameters. For the first The first observation by the road camera 3D world coordinates of a point The backprojection confidence is derived from the depth confidence map. Interpolation is obtained;

[0200] (6e4) The unit response values, representing the total contribution intensity of each 3D lane point to the BEV response map of 1, are weighted and accumulated into the BEV response map R, realizing the mapping from a sparse point set to a dense lane existence probability distribution, and thus obtaining the BEV lane response values. :

[0201] ;

[0202] in, Indicates the first The original response value of the nearest neighbor grid center to this 3D lane point; Indicates the allocation of weights;

[0203] (6f) Normalize and smooth the BEV response map R to obtain a high-density locally enhanced BEV lane representation. .

[0204] Step (7) specifically includes the following steps in sequence:

[0205] (7a) The V2X cooperative perception system is a decentralized, geometrically consistent multi-vehicle cooperative optimization framework that does not rely on high-precision maps or global positioning. The V2X cooperative perception system includes a local perception unit, a V2X communication interface, and a cross-vehicle geometric alignment module. The local perception unit is configured in each participating vehicle and outputs locally estimated unified ground plane parameters and locally reconstructed 3D lane candidate point set. All data are represented in the vehicle's own coordinate system. The V2X communication interface broadcasts the perception results of the vehicle to neighboring vehicles through DSRC dedicated short-range communication and C-V2X cellular vehicle networking standard protocol, and receives corresponding data from other vehicles to form a multi-view observation set. The input of the cross-vehicle geometric alignment module comes from the local results of M vehicles, then constructs a ground plane consistency map, calculates the maximum common subgraph, and jointly optimizes the ground plane parameters.

[0206] In practical deployments, the data quality from vehicle sensors and environmental conditions significantly impact the quality of nodes and edges. Therefore, to improve system robustness, data preprocessing and outlier filtering mechanisms are typically introduced to ensure data accuracy.

[0207] (7b) In the V2X cooperative perception system, the 3D lane point set of each vehicle v is modeled as a structured graph. Node set It includes all detected 3D lane points and their confidence scores, while the edge set... This represents the topological relationship between 3D lane points that belong to the same lane and are spatially adjacent or have a continuous set relationship;

[0208] Node set For the first 3D lane points, For reliability confidence; edge set This indicates the topological connection between lane points. If two points belong to the same lane and have adjacent numbers, then there is an edge connection.

[0209] (7c) Reliability confidence Joint estimation of quality from multi-source sensing:

[0210] ;

[0211] in, Indicates image restoration The input is the classification score output by a lightweight lane confidence estimation module, which uses the ultrafast structure-aware deep lane detection model UFLD. This represents the radar depth confidence level for the corresponding image location. This is a 3D to 2D projection function. For fusion weights;

[0212] Confidence The calculation formula combines visual information and radar depth information, and adjusts parameters... This can balance the impact of two different types of information on the final confidence level. Furthermore, considering the differences in the performance of various sensor types in different scenarios, dynamic adjustments may be necessary based on the specific application scenario. The value of radar information. For example, in low-light environments, camera performance may degrade, and increasing the weight of radar information in such cases could improve overall performance.

[0213] (7d) An uncertainty-aware maximum common subgraph matching algorithm is adopted, which considers both geometric similarity and node uncertainty to address local view differences caused by sensor errors or environmental factors. Since the relative position and pose changes between vehicles also significantly affect the matching results, accurate estimation of the relative pose transformation matrix between vehicles is crucial. This is key to achieving efficient matching. Furthermore, as the number of vehicles increases, the scalability and real-time performance of the algorithm also become important considerations.

[0214] For any two cars and Perform uncertainty-aware maximum common subgraph matching to maximize the geometric and topological consistency of the matched subgraphs;

[0215] (7e) Define node pairs Weighted geometric similarity for:

[0216] ;

[0217] in, For vehicles arrive The relative pose transformation matrix is ​​obtained by solving the global navigation satellite system shared by V2X; The geometric tolerance scale; The vehicle The Reliability confidence level of each lane node; Indicates vehicle The Reliability confidence level of each lane node; This represents the natural exponential function. Indicates vehicle The Middle The coordinates of each lane point Indicates vehicle The Middle The coordinates of each lane point;

[0218] Weighted geometric similarity, by combining geometric distance and confidence, can effectively improve matching accuracy, and the Gaussian kernel function can well adapt to geometric biases at different scales. Choosing the wrong value can lead to overly strict matching, missing many valid matches, while choosing too high a value can increase false matches.

[0219] (7f) To ensure the reasonableness of the matching results, topological constraints require that the matched nodes not only be spatially close but also maintain their original connectivity to preserve global consistency. The matching problem is modeled as a combinatorial optimization with topological constraints:

[0220] ;

[0221] ;

[0222] ;

[0223] in, Represents a binary matching variable. Indicates vehicle The number of candidate lane points, Indicates vehicle The number of candidate lane points;

[0224] If the vehicle Middle node and They are adjacent, that is ( , () is a vehicle Lane map One of the edges in, and ( , Vehicles were matched separately. nodes and ,So and It must also be in the vehicle In the lane map, adjacent lanes must be matched to ensure that the adjacency relationship between lane points is consistent.

[0225] (7g) The alignment problem of lane point sets between vehicles is modeled as a maximum common subgraph optimization problem. Its objective is to maximize the weighted similarity of matching node pairs while maintaining the consistency of the graph topology. The maximum common subgraph optimization problem can be formalized as a maximum weighted subgraph isomorphism problem defined on a bipartite graph with higher-order structure constraints, where the two parts of the bipartite graph are the vehicles and the lane points. and The lane nodes are identified; the Hungarian algorithm is used to solve for the optimal matching pair set. ;

[0226] (7h) Based on the set of optimal matching pairs Align and merge the 3D lane points of multiple vehicles to obtain a globally consistent 3D lane map.

[0227] Step (8) specifically includes the following steps in sequence:

[0228] (8a) The complete end-to-end network model is trained end-to-end using the total loss function, wherein the total loss function is... Relying solely on 2D lane line annotations, the expression is:

[0229] ;

[0230] in, Loss due to 2D lane detection monitoring; , These are all preset hyperparameters used to balance the weights of various losses; For 3D curvature smoothing constraint loss; This represents the lane topology consistency loss.

[0231] (8b) 3D curvature smoothing constraint loss The calculation formula is:

[0232] ;

[0233] Where K is the total number of lanes predicted in the current frame; For the first The number of 3D points on each lane; For the first The first lane The coordinates of a 3D point; Represents the L2 norm;

[0234] (8c) Lane topology consistency loss The calculation formula is:

[0235] ;

[0236] Where y is a predefined set of vertical sampling positions; This represents the number of sampling locations; For the first The lateral coordinates of each lane at the longitudinal distance Y=y are obtained through interpolation; Var( () indicates variance operation;

[0237] (8d) Introducing a new alignment loss To ensure a high-fidelity BEV lane response map R and the final output adaptive resolution BEV lane feature map. The consistency is calculated using the following formula:

[0238] ;

[0239] in, This indicates the vertical resolution of the BEV mesh. This indicates the lateral resolution of the BEV grid. Indicates the vertical grid row index, Indicates the horizontal grid column index. Indicates the lane response value of the BEV. This indicates a BEV lane with high-density, locally enhanced designation.

[0240] (8e) will Add to the total loss function to obtain the updated total loss function. :

[0241] ;

[0242] in, It is a new preset hyperparameter used to control the degree of influence on its loss;

[0243] (8f) Using the updated total loss function Joint optimization of the complete end-to-end network model improves the accuracy and geometric robustness of 3D lane line reconstruction on non-planar roads, even when using only 2D lane annotations.

[0244] In summary, this invention effectively suppresses the degradation effects of adverse weather conditions such as rain, snow, and fog on visual input by introducing a lightweight image restoration network, significantly improving the quality of front-end perception and providing a clear and reliable image foundation for subsequent lane detection. It also solves the problem of drastic performance degradation in low-visibility scenarios using traditional methods. Furthermore, it proposes an explicit ground plane parameter estimation mechanism that accurately characterizes non-planar structures such as curve superelevation and longitudinal slopes, overcoming the 3D reconstruction distortion caused by the assumption of a horizontal road plane in existing methods and significantly improving geometric consistency in complex road scenarios. Finally, it designs a self-supervised multimodal fusion and V2X collaborative optimization framework. Relying solely on 2D lane annotations, it enhances feature robustness through radar-visual complementary fusion and utilizes shared ground plane parameters from multiple vehicles to perform cross-vehicle geometric alignment with the 3D lane map, effectively mitigating issues such as single-vehicle occlusion and sensor noise. This significantly improves the accuracy, completeness, and system-level reliability of 3D lane reconstruction while avoiding reliance on high-precision maps or 3D ground truth annotations, demonstrating strong engineering application value.

[0245] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps, characterized in that: The method includes the following steps in sequence: (1) Acquiring raw sensory images Simultaneously collect radar point cloud data; (2) The original perceptual image An image containing lane line information and affected by adverse weather conditions such as rain, snow, and fog is input into a lightweight image restoration network, which generates a geometrically consistent four-way restored image. ; (3) Construct a multimodal fusion model to restore the four-channel images The corresponding radar point cloud data is input into a multimodal fusion model to obtain a multi-scale feature map F, which generates a depth confidence map aligned with the reconstructed image. ; (4) Based on the multi-scale feature map F and the depth confidence map Obtain unified ground plane parameters; (5) Construct an initial 3D lane candidate point set based on unified ground plane parameters; (6) Map each point in the initial 3D lane candidate point set to the BEV space to generate a high-density locally enhanced BEV lane representation. ; (7) Construct a V2X cooperative perception system, model the initial 3D lane candidate point set of each vehicle into a structured graph with confidence, and output the fused globally consistent 3D lane map; (8) Interpolate and geometrically optimize the globally consistent 3D lane map to finally output structured 3D lane line annotation data.

2. The self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps as described in claim 1, characterized in that: Step (1) specifically refers to: acquiring raw sensory images through four cameras installed in the vehicle. The four cameras respectively cover the front, left, right, and rear views of the vehicle, recording the... The road image is the original perceived image. , H represents the original perceived image. The height, W, represents the original perceived image. The width corresponds to the camera intrinsic parameter matrix. The extrinsic parameter matrix is ,in, For the first Rotation matrix of road cameras; For the first The translation vector of the road camera. Represents the set of rigid body transformations in three-dimensional space; radar point cloud data is acquired simultaneously via vehicle-mounted millimeter-wave radar. , This represents the three-dimensional coordinates of the j-th millimeter-wave radar point in the vehicle coordinate system; This represents the total number of points in a single frame of millimeter-wave radar point cloud; all points are represented in the vehicle coordinate system; raw perception images collected by four cameras. Strict synchronization with radar point cloud data is achieved through hardware triggering or timestamp alignment. The synchronization is achieved through CAN bus timestamp alignment, with a time error of less than 50ms.

3. The self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps as described in claim 1, characterized in that: Step (2) specifically includes the following steps in sequence: (2a) The lightweight image restoration network adopts an encoder-decoder architecture, which includes a shallow encoder with shared weights, a lightweight feature fusion module, a Transformer encoder, an asymmetric Transformer decoder, an upsampling projection head, and a geometric consistency constraint output layer. The shared-weight shallow encoder consists of three convolutional blocks, each containing a 3×3 convolutional layer, batch normalization, and a ReLU activation function, used to extract data from various raw perceptual images. Extracting multi-scale low-dimensional features; The lightweight feature fusion module adaptively enhances the feature response of image regions severely degraded by rain, snow, and fog by introducing a channel attention mechanism. The Transformer encoder divides the input into non-overlapping image blocks and embeds them into a token sequence. Deep features are extracted through L-level backbone blocks, and an in-chip Transformer block is inserted after each level. The asymmetric Transformer decoder uses three learnable weather type query embeddings. As input, contextual information is extracted from the output of the shallow encoder through cross-attention: ； in, For decoder output; For cross-attention functions; Represents a weather query matrix; This represents the key matrix output by the shallow encoder. , This is the output of a shallow encoder. This represents the projection matrix that projects the shallow encoder features onto the key space; This represents the value matrix output by the shallow encoder. , This represents the projection matrix that projects the features of the shallow encoder onto the value space; The upsampling projection head fuses the decoder output with features from each stage of the shallow encoder through skip connections, and then gradually upsamples them to the original resolution via transposed convolution to generate four-channel restored images. ; Geometric consistency constraints on the output layer for reconstructing the image Introducing cross-viewpoint photometric consistency loss, for any pair of adjacent cameras with overlapping fields of view, the known camera extrinsic parameters are used to... Road restoration image projection to the first The image plane is used to generate a composite image and compare it with the first image plane. Actual road reconstruction image Pixel-level comparison is performed in overlapping areas to constrain the restoration process from distorting the multi-view geometry; (2b) Original perceived image First, a true lane line image is obtained through color thresholding, mask generation, and bitwise AND operation; then, edge images are obtained through Canny edge detection. Finally, a combined image integrating lane line semantics and geometric edge information is generated through bitwise OR operations. : ； in, Indicates channel factorial; Indicates bitwise OR; Represents the semantic mask for lane lines; (2c) The original perceptual image As the main input, after normalization, it is divided into... There are three non-overlapping image patches, each of size P*P, where H represents the original perceptual image. The height, W, represents the original perceived image. The width of the image patch is P, where P represents the side length of the image patch. Each image patch is flattened and subjected to a learnable linear projection to obtain the initial token sequence. : ； in, For embedding matrix, For position encoding, For sequence length, For embedded dimensions, For the first The original representation of the input token sequence; (2d) will The input is a Transformer encoder with L stacked layers; the output of the Lth stage is: ； ； Among them, MSA stands for Multi-Head Self-Attention, LN stands for Layer Normalization, and MLP stands for Two-Layer Fully Connected Network. Indicates the first -1 layer output; This represents the intermediate output after self-attention; Indicates the first Layer output; After each stage output, an in-chip multi-head self-attention is inserted to obtain a feature representation that enhances the spatial context: ； in, Represents the query matrix. Represents the key matrix, Represents a value matrix, This represents the dot product of the query and the key. Indicates the scaling factor; -MSA indicates multi-head self-attention within the film.

4. The self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps as described in claim 1, characterized in that: Step (3) specifically includes the following steps in sequence: (3a) Construct a multimodal fusion model, which includes a ResNet visual encoder, a radar depth confidence map generation module and a cross-modal adaptive fusion module, wherein the ResNet visual encoder is composed of a lightweight convolutional neural network with shared weights; the radar depth confidence map generation module includes a point cloud preprocessing unit and a projection rendering unit. The cross-modal adaptive fusion module includes a channel-space gating unit, which includes a 1×1 convolutional layer and a weighted modulation unit. The weighted modulation unit includes a channel weight generation branch, a spatial weight generation branch, and a dual modulation module. (3b) For each restored image, multi-scale visual features are extracted using the ResNet visual encoder in the multimodal fusion model to obtain a multi-scale feature map F: For each restored image Multi-scale feature maps F are extracted using a ResNet visual encoder. , and These represent the restored images. Height and width; (3c) Simultaneously, the radar point cloud data is projected onto the image plane corresponding to each camera to generate a depth confidence map aligned with the restored image. Radar point cloud data constitutes a radar point cloud data set. ,Will Project to the Generate depth confidence maps from image planes. , This indicates the height of the depth confidence map. This represents the width of the depth confidence map; where, Indicates the total number of radar point clouds; Let be the three-dimensional coordinates of the j-th millimeter-wave radar point in the vehicle coordinate system.

5. The self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps according to claim 1, characterized in that: Step (4) specifically includes the following steps in sequence: (4a) Depth confidence map Channel concatenation with the multi-scale feature map F yields a fused feature map. : ； In the formula, This indicates a splicing operation at the channel level; Indicates from the first Road restoration image Multi-scale feature maps extracted from them; Indicates the first Depth confidence map from the perspective of a road camera; C represents the number of visual channels; Indicates the height of the fused feature map; Indicates the width of the fused feature map; (4b) Fusion feature map Global average pooling is performed, and the horizon normal vector is predicted using a lightweight regression head MLP. and offset , where the horizon normal vector From longitudinal slope angle and lateral super high angle Parameterization yields: ； ； ； in, Indicates a local ground plane; ; For any three-dimensional point in the world coordinate system; Indicates the offset; The input feature vector is represented by MLP, which stands for Multilayer Perceptron. Indicates the longitudinal tilt component. Indicates the principal component in the vertical direction. Indicates the lateral tilt component; (4c) Calculate the attention weights of the four-way estimation Weighted fusion yields unified ground plane parameters: ； ； In the formula, Indicates the longitudinal slope angle. Indicates the lateral elevation angle. Indicates the first The longitudinal slope angle estimated by each camera. Indicates the first The estimated lateral elevation angle from each camera. Indicates the first The estimated weights; and Consolidate unified ground plane parameters; The representation of the first The characteristics of road cameras, This represents a learnable query vector. Indicates the calculation of the first The dot product of path features and global query.

6. The self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps according to claim 1, characterized in that: Step (5) specifically includes the following steps in sequence: (5a) From the combined image Extract salient lane point set ,Pick The center of the connected domain; where, Indicates the first Street camera The pixel coordinates of a significant lane point Horizontal pixel coordinates; Vertical pixel coordinates; Indicates the first The number of salient lane points extracted from the road image. A salient lane point is a point that simultaneously satisfies the following conditions: it is located within the semantic region of the lane line, has a strong local geometric edge response, and is selected as a representative center point of the local structure in the connected component analysis. This represents the semantic mask for lane lines. Represents the edge image; (5b) For each point Construct a normalized line of sight : ； in, For the first The intrinsic parameter matrix of the road camera; Calculating the scale factor of the intersection point of a ray and a plane using geometric back projection : ； in, It is the normal vector of the ground plane; Indicates the offset; For the first Rotation matrix of road cameras; For the first The translation vector of the road camera; Solving for 3D world coordinates: ； in, Indicates the first The first observation by the road camera 3D world coordinates of a point; Repeat steps (5a) and (5b) for all 2D lane points to obtain the set of 3D world coordinates, which is the initial 3D lane candidate point set. .

7. The self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps as described in claim 1, characterized in that: Step (6) specifically includes the following steps in sequence: (6a) Construct a slope-aware adaptive resolution BEV map based on terrain slope information, and define the physical spatial range of the BEV map as follows: rice, rice, The horizontal direction of the vehicle coordinate system. This represents the longitudinal direction of the vehicle coordinate system; Represents the maximum horizontal distance. =15; Represents the maximum vertical distance. =50; (6b) Set the base grid resolution for distance perception: along the longitudinal Y-axis, the base grid resolution decreases as distance increases; Specifically, the vertical range Divided into U intervals, the vertical resolution of the u-th interval is ,u∈U, Let be the center distance of the u-th interval. and All are linear coefficients controlling the resolution decay rate; the horizontal X-axis uses a fixed base resolution. rice; (6c) Fuse feature maps Using the camera extrinsic matrix and unified ground plane parameters, the image is projected onto the BEV space through a differentiable view transformation, and cross-view aggregation is provided to generate dense BEV feature maps. ; A spatial prior module is constructed, comprising a geometric deviation map generation module and a weighted prediction head. The geometric deviation map generation module calculates the signed distance from the world coordinates of the center point of each BEV cell to the ground plane, generating a geometric deviation map. This geometric deviation map is then compared with... The weight prediction head consists of a 3×3 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a sigmoid function connected in sequence, mapping the fused features to an attention heatmap; the input to the spatial prior module is a dense BEV feature map. Unify ground plane parameters and offset d, and output a BEV spatial weight map. ,in, This represents the height of the BEV spatial weight map, i.e., the vertical resolution. This represents the width of the BEV spatial weight map, i.e., its horizontal resolution. (6d) Perform slope- or curvature-aware dynamic resolution enhancement on top of the base mesh: For each base mesh cell in the BEV space, double the local mesh resolution of each base mesh cell in the BEV space if any of the following three conditions are met, i.e. , ,in, Indicates adaptive horizontal resolution. Indicates adaptive vertical resolution. Indicates the basic horizontal resolution. This represents the basic vertical resolution; the three conditions include condition 1, condition 2, and condition 3. Condition 1: Attention weights corresponding to basic grid cells ,in The preset threshold; Condition 2: Absolute value of the longitudinal slope angle of the road ,in ; Condition 3: Absolute value of the road's lateral superelevation angle ,in ; in, Indicates the longitudinal slope sensitivity threshold. This indicates an extremely high sensitivity threshold in the horizontal direction; (6e) Based on the adaptive resolution network at different distances, a diffusion feature allocation is performed on the initial 3D lane candidate point set to construct a high-fidelity BEV lane response map R; (6f) Normalize and smooth the BEV response map R to obtain a high-density locally enhanced BEV lane representation. .

8. The self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps according to claim 1, characterized in that: Step (7) specifically includes the following steps in sequence: (7a) The V2X cooperative perception system includes a local perception unit, a V2X communication interface and a cross-vehicle geometry alignment module; the local perception unit is configured in each participating vehicle, and outputs locally estimated unified ground plane parameters and locally reconstructed 3D lane candidate point set, with all data represented in the vehicle's own coordinate system. The V2X communication interface broadcasts the vehicle's perception results to nearby vehicles through DSRC dedicated short-range communication and C-V2X cellular vehicle networking standard protocol, and receives corresponding data from other vehicles to form a multi-view observation set. The cross-vehicle geometry alignment module takes local results from M vehicles as input, then constructs a ground plane consistency map, calculates the maximum common subgraph, and jointly optimizes the ground plane parameters. (7b) In the V2X cooperative perception system, the 3D lane point set of each vehicle v is modeled as a structured graph. Node set It includes all detected 3D lane points and their confidence scores, while the edge set... This represents the topological relationship between 3D lane points that belong to the same lane and are spatially adjacent or have a continuous set relationship; Node set For the first 3D lane points, For reliability confidence; edge set This indicates the topological connection between lane points. If two points belong to the same lane and have adjacent numbers, then there is an edge connection. (7c) Reliability confidence Joint estimation of quality from multi-source sensing: ； in, Indicates image restoration The input is the classification score output by a lightweight lane confidence estimation module, which uses the ultrafast structure-aware deep lane detection model UFLD. This represents the radar depth confidence level for the corresponding image location. This is a 3D to 2D projection function. For fusion weights; (7d) For any two vehicles and Perform uncertainty-aware maximum common subgraph matching to maximize the geometric and topological consistency of the matched subgraphs; (7e) Define node pairs Weighted geometric similarity for: ； in, For vehicles arrive The relative pose transformation matrix is ​​obtained by solving the global navigation satellite system shared by V2X; The geometric tolerance scale; The vehicle The Reliability confidence level of each lane node; Indicates vehicle The Reliability confidence level of each lane node; This represents the natural exponential function. Indicates vehicle The Middle The coordinates of each lane point Indicates vehicle The Middle The coordinates of each lane point; (7f) Model the matching problem as a combinatorial optimization with topological constraints: ； ； ； in, Represents a binary matching variable. Indicates vehicle The number of candidate lane points, Indicates vehicle The number of candidate lane points; If the vehicle Middle node and They are adjacent, that is ( , () is a vehicle Lane map One of the edges in, and ( , Vehicles were matched separately. nodes and ,So and It must also be in the vehicle In the lane map, adjacent lanes must be matched to ensure that the adjacency relationship between lane points is consistent. (7g) The alignment problem of lane point sets between vehicles is modeled as a maximum common subgraph optimization problem. Its objective is to maximize the weighted similarity of matching node pairs while maintaining the consistency of the graph topology. The maximum common subgraph optimization problem can be formalized as a maximum weighted subgraph isomorphism problem defined on a bipartite graph with higher-order structure constraints, where the two parts of the bipartite graph are the vehicles and the lane points. and The lane nodes are identified; the Hungarian algorithm is used to solve for the optimal matching pair set. ; (7h) Based on the set of optimal matching pairs Align and merge the 3D lane points of multiple vehicles to obtain a globally consistent 3D lane map.

9. The self-supervised multimodal fusion and collaborative optimization method applicable to curves and ramps according to claim 1, characterized in that: Step (8) specifically includes the following steps in sequence: (8a) In the BEV space, interpolation and geometric optimization are performed on the globally consistent 3D lane map. A complete end-to-end network model, including a lightweight image restoration network, a multimodal fusion model, and a V2X cooperative perception system, is trained using a total loss function that relies solely on 2D supervision. Relying solely on 2D lane line annotations, the expression is: ； in, Loss due to 2D lane detection monitoring; , These are all preset hyperparameters used to balance the weights of various losses; For 3D curvature smoothing constraint loss; This represents the lane topology consistency loss. (8b) 3D curvature smoothing constraint loss The calculation formula is: ； Where K is the total number of lanes predicted in the current frame; For the first The number of 3D points on each lane; For the first The first lane The coordinates of a 3D point; Represents the L2 norm; (8c) Lane topology consistency loss The calculation formula is: ； Where y is a predefined set of vertical sampling positions; This represents the number of sampling locations; For the first The lateral coordinates of each lane at the longitudinal distance Y=y are obtained through interpolation; Var( () indicates variance operation; (8d) Introducing a new alignment loss To ensure a high-fidelity BEV lane response map R and the final output adaptive resolution BEV lane feature map. The consistency is calculated using the following formula: ； in, This indicates the vertical resolution of the BEV mesh. This indicates the lateral resolution of the BEV grid. Indicates the vertical grid row index, Indicates the horizontal grid column index. Indicates the lane response value of the BEV. This indicates a BEV lane with high-density, locally enhanced designation. (8e) will Add to the total loss function to obtain the updated total loss function. : ； in, It is a new preset hyperparameter used to control the degree of influence on its loss; (8f) Using the updated total loss function The complete end-to-end network model is jointly optimized to output structured 3D lane line annotation data, thereby improving the accuracy and geometric robustness of 3D lane line reconstruction on non-planar roads using only 2D lane annotations.

Images