A scene-adaptive spatiotemporal coordination lightweight three-dimensional lane detection method, training method and system, electronic equipment and storage medium

Abstract

The application provides a scene-adaptive spatio-temporal collaborative lightweight three-dimensional lane detection method, a training method, a system, an electronic device and a storage medium, steps of which comprise: extracting multi-scale image features of a current frame and historical frames, projecting three-dimensional reference anchor points to the current frame and sampling, predicting coordinate offsets to obtain candidate coarse lane reference points; predicting candidate coarse three-dimensional lanes and multi-attribute scene vectors; predicting inter-frame relative poses and performing three-dimensional transformation on historical candidate coarse lane reference points; transferring historical candidate coarse three-dimensional lanes to the current frame according to a cross-frame correlation matrix; correcting lane features of the historical frames in a time window and reading, and outputting fused lane queries; constructing parameter anchor points to interpolate candidate coarse three-dimensional lanes, predicting coordinate residuals to obtain fine curve support points, and outputting fine three-dimensional lane coordinates. The application also provides a corresponding training method, a system, an electronic device and a readable storage medium. The application improves detection stability under curved roads, slopes, occlusions and camera disturbances, while maintaining lightweight and adapting to real-time deployment on vehicles.

Description

Technical Field

[0001] This invention relates to the technical fields of autonomous driving environment perception, computer vision, 3D geometric reconstruction and temporal feature modeling, and in particular to a lightweight 3D lane detection method, training method, system, electronic device and storage medium with scene adaptive spatiotemporal coordination. Background Technology

[0002] 3D lane detection is used to output the lateral position, longitudinal extension, height variation, visibility, and semantic attributes of lane lines in a vehicle coordinate system or a road coordinate system. It is an important perceptual foundation for lane keeping, path planning, vehicle lateral control, and road topology understanding. Compared with 2D lane detection, which only outputs lane pixel coordinates in the image plane, 3D lane detection also needs to deal with monocular scale constraints, road slope and undulation, long-distance resolution degradation, complex curve representation, and video temporal stability.

[0003] Existing 3D lane detection methods based on sparse queries or Transformer decoders typically describe lanes using a fixed number of uniformly distributed longitudinal sampling points. While a fixed tensor dimension facilitates batch processing, it fails to allocate representational power reasonably based on distance, curvature, slope, and visibility, easily leading to undersampling at long distances, high-curvature curves, and undulating roads. Some methods use a single polynomial or a single sample line to represent the entire lane, which struggles to simultaneously maintain local shape preservation, overall smoothness, and continuous completion of occluded sections in scenarios involving local lane marking wear, long-section occlusion, and lane splitting or merging.

[0004] To leverage video continuity, existing methods can transfer features or detection results from historical frames to the current frame. However, vehicle motion, camera vibration, nominal mounting extrinsic parameter residuals, and inter-frame motion estimation errors can cause inconsistencies between historical information and the spatial location of the current frame. Directly accumulating historical image features or query features before alignment can lead to lane ghosting, lateral drift, and error accumulation. Using pose transformation and 3D coordinates alone cannot automatically resolve the cross-frame lane identification problem, as different lane queries in historical frames may be swapped with queries in the current frame. Therefore, cross-frame lane associations need to be established after coordinate alignment.

[0005] Traditional bidirectional temporal networks typically read both past and future frames simultaneously, making them unsuitable for online driving perception that requires immediate output of the current frame. While unidirectional smoothing using only historical results satisfies causality, it fails to leverage current observations to suppress historical errors. Therefore, it is necessary to establish bidirectional information exchange between the current and historical frames within a causal window that includes only the current and historical frames, and to clearly define the data flow order of this bidirectional exchange.

[0006] Furthermore, attributes such as straight road lines, curves, slopes, intersections, tunnels, low illumination, and glare are not mutually exclusive. Using mutually exclusive scene classification would result in the loss of simultaneously existing road geometry and lighting attributes. The applicability of geometric priors such as lane smoothness, adjacent lane direction consistency, lane width, and topology also varies across different scenarios. Fixed prior weights lack scene adaptability, while completely free learnable weights may approach zero, rendering the priors ineffective. Summary of the Invention

[0007] To address the technical problems of existing lane detection methods, such as fixed sampling, missing mapping, historical misalignment, bidirectional dependence, single basis function, generation lag, mixed training and detection, and weight degradation, this invention proposes a lightweight 3D lane detection method, training method, system, electronic device, and storage medium with scene-adaptive spatiotemporal collaboration. Under the condition of clear coordinates and scale benchmarks, this invention first generates a coarse 3D lane, coordinate uncertainty, and scene attributes from the current frame, then estimates the relative pose between frames and performs coordinate and feature alignment. After establishing cross-frame lane association, it performs causal two-layer temporal interaction and further maps non-uniform parameter positions into computable fine 3D support points. Finally, it outputs a complete scheme of fine 3D lane through local multi-basis function soft gating, which can improve the detection stability in scenes with curves, slopes, occlusion, and camera disturbances without reading future frames.

[0008] To achieve the above objectives, the technical solution of the present invention is implemented as follows: a lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination, comprising the following steps:

[0009] S1: Extract multi-scale image features of the current frame and multi-scale image features of historical frames through a lightweight backbone network; set a three-dimensional reference anchor point in the vehicle coordinate system and project it onto the current frame image; sparsely sample image features at the projection position and predict the three-dimensional coordinate offset to obtain candidate coarse lane reference points.

[0010] S2: Perform coarse lane prediction and scene attribute prediction in parallel on the multi-scale image features of the current frame, and output candidate coarse 3D lanes and multi-attribute scene vectors respectively;

[0011] S3: Based on the multi-scale image features of the current frame and the multi-scale image features of the historical frames, predict the relative pose between frames, perform a three-dimensional transformation on the historical candidate coarse lane reference points, and output the transformed historical candidate coarse three-dimensional lanes.

[0012] S4: Construct a cross-frame association matrix based on the current candidate coarse 3D lanes and the transformed historical candidate coarse 3D lanes. Based on the cross-frame association matrix, pass the reference point visibility, lane semantic probability and query embedding in the historical candidate coarse 3D lanes to the current frame.

[0013] S5: Within the causal time-series window, correct the associated historical frame lane features using the current frame lane features; generate historical frame time-series weights based on historical support confidence and time-series attenuation coefficients; read the corrected historical frame lane features using the current frame lane query based on the historical frame time-series weights, and output the fused lane query through gated residual fusion.

[0014] S6: Construct non-uniformly distributed parameter anchor points, interpolate candidate coarse 3D lanes at the parameter anchor points to obtain initial 3D points; generate parameter position codes based on the positions of the parameter anchor points, project the initial 3D points onto the current frame image sampling to obtain image sampling features, combine the parameter position codes, image sampling features and the fused lane query prediction 3D coordinate residuals to obtain fine curve support points;

[0015] S7: Construct multiple piecewise curve basis functions using fine curve support points, and then weight and fuse the outputs of the multiple piecewise curve basis functions to output fine three-dimensional lane coordinates in the vehicle coordinate system.

[0016] The method for setting three-dimensional reference anchor points in the vehicle coordinate system and projecting them onto the current frame image is as follows:

[0017] ;

[0018] in, For the first Candidate coarse three-dimensional lane number The projected pixel coordinates of each 3D reference anchor point 3D reference anchor point homogeneous three-dimensional reference anchor points, For the camera intrinsic parameter matrix, Install the extrinsic parameter matrix in the nominal direction from the camera coordinate system to the vehicle coordinate system. To normalize homogeneous coordinates;

[0019] The method for sparsely sampling image features at the projection location is as follows:

[0020] For each projected pixel coordinate Based on multi-scale image features of the current frame and historical frames, using projected pixel coordinates Bilinear interpolation sampling is performed around the center to obtain multi-scale features;

[0021] The candidate rough lane reference point for: ;in, This represents the three-dimensional coordinate offset.

[0022] The method for performing coarse lane prediction and scene attribute prediction in parallel on the multi-scale image features of the current frame, and outputting candidate coarse 3D lanes and multi-attribute scene vectors respectively, is as follows:

[0023] Set a fixed number of lane queries on the multi-scale image features of the current frame; predict the lane existence, reference point visibility and lane semantic probability of each lane query based on the multi-scale image features of the current frame.

[0024] For each lane query, project pixel coordinates Multi-scale features obtained from sampling Combined with sinusoidal position coding and query embedding Constructing input features for the coordinate prediction branch for:

[0025] ;

[0026] Input features Input coordinate prediction branch, synchronously output the mean 3D coordinates of each candidate rough lane reference point and the logarithmic variance of three-dimensional coordinates for: ;in, For coordinate prediction head network;

[0027] Logarithmic variance of three-dimensional coordinates Determine the uncertainty of coordinate regression for:

[0028]

[0029] in, For spatial dimension indexing, These represent the three coordinate axes in three-dimensional space. For candidate rough lane reference point number Log-variance on 3D coordinates;

[0030] Candidate coarse 3D lanes are constructed by considering lane existence, reference point visibility, lane semantic probability, query embedding, 3D coordinate mean, and coordinate regression uncertainty.

[0031] The deepest image features of the current frame are analyzed through the scene attribute branch. Perform global average pooling and fully connected hidden layer mapping to output multidimensional raw logical values. for: ;in, For the current frame index, For global average pooling; A head network for classifying scene attributes;

[0032] For multidimensional primitive logical values Applying a Sigmoid activation function to each dimension, the output is a multi-attribute scene vector. for: .

[0033] The method for predicting the inter-frame relative pose based on the multi-scale image features of the current frame and the multi-scale image features of historical frames, performing a 3D transformation on the historical candidate coarse lane reference points, and outputting the transformed historical candidate coarse 3D lanes is as follows:

[0034] Input the multi-scale image features of the current frame and the multi-scale image features of the historical frames into the pose prediction branch, and output the inter-frame relative pose transformation from the historical frames to the current frame. ;

[0035] Based on inter-frame relative pose transformation A three-dimensional transformation is performed on the historical candidate coarse lane reference points to obtain the transformed historical candidate coarse lane reference points: ;in, For historical frame indexing, For historical frames Candidate coarse lane reference points for candidate coarse 3D lanes The transformed historical candidate coarse lane reference point;

[0036] The current candidate crude lane reference point By mapping the inverse inter-frame relative pose transformation from the current frame to the historical frame onto the historical frame image plane, the projected pixel coordinates of the historical frame are obtained. for: ;in, This represents the inverse inter-frame relative pose transformation from the current frame to historical frames. ;

[0037] Projecting pixel coordinates in historical frames At this point, bilinear interpolation sampling is performed on the multi-scale image features of historical frames to obtain the multi-scale features of historical frames.

[0038] The method for constructing a cross-frame correlation matrix based on the current candidate coarse 3D lanes and the transformed historical candidate coarse 3D lanes is as follows:

[0039] Cost of constructing cross-frame associations:

[0040] ;

[0041] in, For the current number Candidate coarse three-dimensional lanes and historical first The cross-frame association cost of candidate coarse 3D lanes, For ordered curve distances, For lane semantic probability distance, To query the embedding distance, These are the weight coefficients for ordered curve distance, lane semantic probability distance, and query embedding distance, respectively. For the current number Candidate coarse lane reference points for candidate coarse 3D lanes For the transformed history Candidate coarse lane reference points for candidate coarse 3D lanes For the current number The semantic probability of each candidate coarse 3D lane. For the first time in history The semantic probability of each candidate coarse 3D lane. For the current number Query embedding of candidate coarse 3D lanes, For the first time in history Query embedding of candidate coarse 3D lanes;

[0042] Cross-frame association cost Constructing the cross-frame association cost matrix A cross-frame correlation matrix is ​​generated by Sinkhorn normalization. for: ,in, For temperature coefficient, This is the normalization function for a double random matrix.

[0043] The method of correcting and associating historical frame lane features with current frame lane features; generating historical frame time-series weights based on historical support confidence and time-series attenuation coefficients; reading corrected historical frame lane features using current frame lane queries based on historical frame time-series weights; and outputting fused lane queries through gated residual fusion is as follows:

[0044] Using the current frame lane features as keys and values, and the associated historical frame lane features as queries, cross-attention is performed to obtain the corrected historical frame lane features:

[0045] ;

[0046] in, For the lane features of the associated historical frames, This is the cross-attention output obtained by using historical frame lane features as the query and current frame lane features as the key and value. Lane features in the corrected historical frames;

[0047] Calculate historical support confidence for: ;in, For pose alignment confidence. For local feature similarity, For the visibility of the reference point, For confidence prediction networks;

[0048] Calculate the normalized historical frame temporal weights for:

[0049] ;

[0050] in, For projection and correlation validity mask, A positive time-series decay coefficient. It is the minimum constant. For the first The original temporal weights of each historical frame;

[0051] Using the current frame lane query as the query and the corrected historical frame lane features as the key and value, cross-attention is performed to obtain the lane query after overlaying historical information. for:

[0052] ;

[0053] in, For the current frame lane query, The cross-attention output reads the corrected historical frame lane features for the current frame lane query;

[0054] Gated residual fusion output fusion lane query for:

[0055] ;

[0056] in, For the gated vector, This is for element-wise multiplication.

[0057] The method for constructing non-uniformly distributed parameter anchor points is as follows:

[0058] Based on the local geometric features and coordinate regression uncertainties along the lane direction calculated from the candidate coarse 3D lanes, a local state vector is formed. for: ;in, To normalize the vertical distance, To normalize the local curvature, To normalize the road slope angle, For the visibility probability of the reference point, For coordinate regression uncertainty, To support historical reliability;

[0059] Based on multi-attribute scene vectors Lane integrity and the uncertainty of mean coordinate regression Generate modulation sampling coefficients for: ;in, For modulation sampling coefficient prediction network, It is a positive value mapping function;

[0060] Calculate the importance of sampling for: ;in, This is a bias term for the importance of sampling;

[0061] Importance of sampling Normalization is performed and a cumulative distribution is constructed. The non-uniformly distributed parameter anchor points are then generated through inversion of the cumulative distribution. ;

[0062] The initial three-dimensional points for: ;in, For the set of candidate coarse lane reference points, It is an interpolation function;

[0063] The three-dimensional coordinate residual for: ;in, For residual prediction head network, Encode the parameter position. Image sampling features;

[0064] The fine curve support point for: .

[0065] The method for constructing multiple piecewise curve basis functions using fine curve support points, and then weighting and fusing the outputs of these multiple piecewise curve basis functions to output fine three-dimensional lane coordinates in the vehicle coordinate system is as follows:

[0066] Calculate the local basis function gate weights for: ;

[0067] in, For local visibility, For local coordinate regression uncertainty, For gated prediction networks;

[0068] The fine three-dimensional lane coordinates for: ;in, For the set of basis functions of the curve, For the first Each basis function at the parameter anchor point Gating weights at the location, For the first Each basis function at the parameter anchor point The output at that location.

[0069] A lightweight 3D lane training method with scene-adaptive spatiotemporal coordination, characterized by comprising:

[0070] Construct a bipartite graph matching cost between the predicted 3D lanes and the actual 3D lanes in the current frame to complete lane-level matching;

[0071] Based on the matching results, the detection loss, attribute loss, temporal loss and regularization loss are calculated, and the total loss is obtained by weighted summation.

[0072] Based on the total loss, the coordinate prediction branch parameters, scene attribute branch parameters, pose prediction branch parameters, confidence prediction network parameters, modulation sampling coefficient prediction network parameters, residual prediction head network parameters, and gated prediction network parameters in the 3D lane detection method are jointly optimized through backpropagation.

[0073] A lightweight 3D lane detection system with scene-adaptive spatiotemporal coordination, characterized in that it includes:

[0074] The image and camera parameter input module is used to receive temporal road images and camera calibration parameters, and perform image preprocessing;

[0075] The current frame joint coarse prediction module is used to extract multi-scale image features of the current frame, perform coarse lane prediction and scene attribute prediction in parallel, and output candidate coarse 3D lanes and multi-attribute scene vectors.

[0076] The scene attribute modulation module is used to input multi-attribute scene vectors into each modulation unit to achieve scene adaptive adjustment of sampling density, temporal weight and basis function fusion weight;

[0077] The inter-frame pose and sparse projection alignment module is used to predict the inter-frame relative pose, perform 3D transformation on historical candidate coarse lane reference points, and perform sparse projection sampling on the multi-scale image features of historical frames.

[0078] The cross-frame lane association module is used to construct a cross-frame association matrix and establish the correspondence between historical candidate coarse 3D lanes and current candidate coarse 3D lanes.

[0079] The causal dual-layer temporal interaction module is used to correct the lane features of the historical frame with the lane features of the current frame within the causal temporal window, and to read the corrected lane features of the historical frame with the lane query of the current frame, and output the fused lane query through gated residual fusion.

[0080] The adaptive parameter anchor point and fine support point generation module is used to construct non-uniformly distributed parameter anchor points, interpolate candidate coarse 3D lanes to obtain initial 3D points, and combine fused lane query, parameter position encoding and image sampling features to predict 3D coordinate residuals to obtain fine curve support points.

[0081] The local multi-basis function reconstruction module is used to construct multiple piecewise curve basis functions, and to perform weighted fusion of the outputs of multiple piecewise curve basis functions to output fine three-dimensional lane coordinates in the vehicle coordinate system.

[0082] The 3D lane output module is used to perform post-processing on the detection results and output standardized 3D lane perception results.

[0083] The training constraint module, enabled only during the training phase, is used to construct the loss function and perform network parameter optimization.

[0084] The beneficial effects of this invention are:

[0085] 1. Higher efficiency of adaptive sampling: Compared with the three-dimensional lane detection method with fixed uniform sampling, the present invention keeps the number of fine points and tensor dimension fixed, and at the same time reallocates parameter anchor points according to coarse lane geometry, visibility, coordinate uncertainty, historical support reliability and scene attributes, so that the limited amount of computation is given priority to long distances, sharp curves, slopes and occluded sections with reliable historical support, thereby improving the representation accuracy of complex road sections with the same amount of computation.

[0086] 2. Complete geometric representation link: Compared with the method that only outputs the position of non-uniform parameters, the present invention further performs coarse curve interpolation and three-dimensional coordinate residual regression to clearly obtain the fine curve support points, so that multiple curve branches have actual calculable three-dimensional inputs, thus completing the technical chain of "non-uniform parameter position - three-dimensional curve support point".

[0087] 3. More robust temporal fusion: Compared with the method of directly accumulating historical features, this invention first estimates the relative pose that satisfies the SE(3) constraint, aligns the historical 3D points and image features, and then establishes cross-frame lane association based on ordered curves, semantics and query embedding, which can significantly reduce the erroneous fusion caused by historical lane identity exchange and spatial misalignment.

[0088] 4. Online causal bidirectional interaction: Compared with traditional bidirectional networks that read future frames, this invention only uses the current frame and historical frames, and utilizes historical context through sequential bidirectional interaction of "current correction history - current reading history", which fully meets the causal requirements of online reasoning for autonomous driving.

[0089] 5. Enhanced curve representation capability: Compared with using only a single global curve gating weight for the entire lane, this invention adopts point-by-point or segment-by-segment local gating, enabling different combinations of curve basis functions to be used for complete sections, local noise sections, and occluded sections. It preserves details at the near end, suppresses noise in the middle section, and smoothly completes at the far end, while taking into account both local morphological fidelity and global continuity.

[0090] 6. Balancing scene adaptability and prior validity: Compared with fixed or unbounded prior weights, this invention outputs non-mutually exclusive multi-attribute scene vectors in advance during the coarse prediction stage and generates geometric prior weights with positive lower bounds and preset upper bounds; topological validity masks are used at intersections or merging / diverting areas to avoid prior weights degenerating to zero and causing prior failure, and to prevent excessive constraints on real topological changes. Attached Figure Description

[0091] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0092] Figure 1 This is a flowchart of the three-dimensional lane detection method of the present invention.

[0093] Figure 2 This is a schematic diagram of the cross-frame lane association and causal two-layer sequential interaction of the present invention.

[0094] Figure 3 This is a schematic diagram illustrating the pose estimation, coordinate transformation, and sparse projection alignment of historical image features according to the present invention.

[0095] Figure 4 This is a schematic diagram of the non-uniform parameter anchor points, fine support points, and local multi-basis function reconstruction of the present invention.

[0096] Figure 5 This is a schematic diagram of the multi-attribute scene modulation and bounded geometric prior training structure of the present invention.

[0097] Figure 6 This is a schematic diagram of the three-dimensional lane detection system of the present invention. Detailed Implementation

[0098] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0099] This invention proposes a lightweight 3D lane detection method, training method, system, electronic device, and storage medium with scene-adaptive spatiotemporal collaboration, aiming to solve the following technical problems: fixed uniform sampling is difficult to adapt to long distances, sharp curves, and slope geometry; only non-uniform parameter positions are generated without mapping from parameter positions to 3D support points; historical information is spatially misaligned under pose perturbation and identity query is unstable across frames; bidirectional temporal networks using future frames are not suitable for online inference; single curve basis functions are difficult to simultaneously consider local morphology, smoothness, and occlusion completion; scene attribute generation order lags behind scene modulation module; training process and online detection process are mixed; and the problem of degeneracy of unbounded geometric prior weights is easy to solve.

[0100] This invention adopts an online detection route of "current frame joint coarse prediction - inter-frame pose estimation and sparse projection - cross-frame lane association - causal two-layer sequential interaction - scene modulation non-uniform parameter anchor points - coarse curve interpolation and 3D residual refinement - local multi-basis function soft gating reconstruction": First, multi-scale features are extracted based on monocular causal image sequences, and coarse 3D lanes, coordinate regression uncertainty and multi-attribute scene vectors are output in parallel; Second, the rigid relative pose of the inter-frame SE(3) is estimated, and 3D coordinate transformation and sparse projection sampling of historical images are completed; Then, cross-frame lane association is established through geometric-semantic-embedding joint cost, and sequential bidirectional temporal interaction of "current correction history - current reading history" is performed within the causal window; Then, non-uniform parameter anchor points are generated by combining scene attributes and geometric confidence, and fine support points are obtained through coarse interpolation and residual regression; Finally, fine 3D lanes are output through local multi-basis function soft gating. This invention effectively improves detection stability and representation accuracy in scenarios involving curves, slopes, occlusion, and camera disturbances without reading future frames under online constraints, while maintaining lightweight computing characteristics and adapting to the real-time deployment requirements of vehicle-mounted terminals.

[0101] To maintain consistency throughout the text, the core symbols are defined as follows:

[0102] Let the current time be The historical window length is The causal image sequence is ,in This represents the road image in the current frame. This represents the camera intrinsic parameter matrix. This represents the nominal mounting extrinsic parameter matrix from the camera coordinate system to the vehicle coordinate system, and The horizontal axis, vertical axis, and forward axis of the vehicle coordinate system are denoted as follows: , and .

[0103] Represents the set of reference points for the rough lane. Indicates the adaptive parameter anchor point. Indicates the support points of the fine curve. This indicates the final fine-grained lane sampling point. Visibility is used... This indicates that coordinate regression uncertainty is used This indicates the use of an effectiveness mask. This indicates that lane integrity is used. express.

[0104] Indicates the current frame number Hierarchical multiscale image feature tensors Indicates the first Frame number Hierarchical multiscale image feature tensors For feature level. Indicates the first Lane number The projected pixel coordinates of a 3D reference anchor point on the image plane. Indicates the first Candidate rough lane number The three-dimensional coordinates of a rough lane reference point. Indicates the first The query embedding vector for each lane has a dimension of .

[0105] Example 1

[0106] This embodiment proposes a lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination. It generates coarse 3D lane, coordinate uncertainty, and multi-attribute scene vectors based on monocular images of the current and historical frames. It predicts the relative pose from historical frames to the current frame, performs coordinate transformation and sparse projection sampling on historical 3D points and image features. After cross-frame lane association, it sequentially performs current information correction of historical information and current query to retrieve historical information within a causal window. Non-uniform parameter anchor points are generated based on coarse lane geometry, visibility, uncertainty, historical support reliability, and scene attributes. Fine support points are formed through coarse curve interpolation and 3D residual regression, and a fine 3D lane is output using local multi-basis function soft gating. This invention can improve detection stability in scenes with curves, slopes, occlusion, and camera disturbances without reading future frames.

[0107] This approach employs a technical route of "joint coarse prediction of the current frame—inter-frame pose estimation and sparse projection—cross-frame lane association—causal two-layer sequential interaction—scene modulation of non-uniform parameter anchors—coarse curve interpolation and 3D residual refinement—local multi-basis function soft-gated reconstruction," utilizing only monocular images of the current and historical frames to achieve refined 3D lane detection without reading future frames under online constraints. The core logic is as follows: First, a lightweight backbone network extracts multi-scale features and outputs coarse lane information, uncertainty, and scene attributes in parallel; then, rigid poses between frames are estimated to achieve spatial alignment and establish cross-frame lane identity association; subsequently, sequential bidirectional temporal interaction is performed within a causal window to fuse historical context; then, non-uniform parameter anchors are generated based on geometry, confidence, and scene attributes, mapped to refined 3D support points; finally, high-precision 3D lanes are obtained through local multi-basis function soft-gated reconstruction. The overall process is as follows: Figure 1 As shown, the specific steps are as follows:

[0108] S1: Extract multi-scale image features of the current frame and multi-scale image features of historical frames through a lightweight backbone network; set a three-dimensional reference anchor point in the vehicle coordinate system and project it onto the current frame image; sparsely sample image features at the projection position and predict the three-dimensional coordinate offset to obtain candidate coarse lane reference points.

[0109] A causal road image sequence consisting of the current frame and at least one historical frame acquired by a vehicle-mounted monocular camera is obtained, along with the intrinsic parameter matrix of the monocular camera and the nominal mounting extrinsic parameter matrix of the camera relative to the vehicle coordinate system. The causal road image sequence is normalized. The vehicle coordinate system is defined with the vehicle's lateral direction as the first coordinate axis, the vertical upward direction as the second coordinate axis, and the vehicle's forward direction as the third coordinate axis. A fixed number of three-dimensional reference anchor points are set in the vehicle coordinate system. The three-dimensional reference anchor points are projected onto the current frame image using the camera's intrinsic parameter matrix and the nominal mounting extrinsic parameter matrix. Multi-scale image features are sampled at the projection positions, and the three-dimensional coordinate offset relative to the three-dimensional reference anchor points is predicted to obtain the coarse lane reference point.

[0110] The input only includes the current and historical frames from the vehicle-mounted monocular camera; future frames after the current frame are not read. Each image, after size normalization, color normalization, and optional brightness perturbation, is input into a lightweight backbone network with shared weights to obtain multi-scale features. and ,in, Indicates the feature level.

[0111] The causal road image sequence is composed of historical frames and the current frame, continuously acquired chronologically by an onboard monocular camera, strictly adhering to the online inference constraint of "only including the past and present, not reading the future." Let the current time be... The historical window length is The sequence is then arranged in chronological order as follows: ,in For cached historical frames, This is the current frame to be detected; the sequence length is... , The optimal value is 2-5, which balances timing gain and computational cost.

[0112] Image normalization preprocessing: Perform uniform preprocessing on all images in the sequence. The process is as follows:

[0113] Size normalization: scaling the original acquired image to a fixed resolution. (e.g., 360×640), bilinear interpolation is used to ensure smooth edges; color normalization: the RGB three-channel pixel values ​​are linearly mapped from [0,255] to the [0,1] interval, and then channel-level standardization is performed according to the mean and standard deviation of the ImageNet dataset. ,in , ; Optional brightness perturbation: During the training phase, a brightness gain within ±15% and a contrast perturbation within ±10% are randomly applied to the image to simulate changes in illumination; This step is turned off during the inference phase.

[0114] The projection operation of the 3D reference anchor point is performed after image normalization. The projection coordinates strictly correspond to the resolution of the normalized image to ensure that the sampling position matches the feature map coordinate system.

[0115] Coarse lane reference point generation logic: Coarse lane reference points are obtained from "3D reference anchor point coordinates + 3D coordinate offset predicted by the network". First, uniformly distributed initial reference anchor points are set in the vehicle coordinate system. The network predicts the 3D offset of each anchor point. The final rough lane reference point satisfies The design uses fixed anchor points as geometric priors to avoid monocular scale ambiguity, while adapting to the real lane geometry through offset regression.

[0116] This invention does not advocate restoring absolute scale under completely unknown camera geometry conditions. Instead, it establishes a scale benchmark using the intrinsic parameters of a fixed-mount camera, nominal extrinsic parameters, 3D reference anchor points in the vehicle coordinate system, and metric 3D lane training annotations. Training annotations are obtained through two methods: first, using publicly available datasets such as Apollo 3D Lane and OpenLane, whose annotations are generated by fitting LiDAR point clouds and manual refinement; second, using a self-built dataset generated by acquiring point clouds from high-precision LiDAR and spatiotemporally registering them with images.

[0117] Set the relevant 3D reference anchor points in the vehicle coordinate system. And projected onto the image plane:

[0118] ;

[0119] in, As a homogeneous three-dimensional reference anchor point, To normalize homogeneous coordinates, This is the intrinsic parameter matrix of a monocular camera. An extrinsic parameter matrix is ​​nominally installed from the camera coordinate system to the vehicle coordinate system. The network performs sparse multi-scale sampling near the projection location and predicts the coordinate offset relative to the 3D reference anchor point, so that the camera parameters actually participate in coarse 3D lane decoding, rather than just being used for subsequent temporal projection.

[0120] Sparse multi-scale sampling implementation: Sparse sampling around the projection position of the 3D reference anchor point reduces the computational load from "full-image pixel level" to "lane point level", meeting the lightweight real-time inference requirements of vehicle-mounted devices. The implementation method is as follows: for each projected pixel coordinate... On the feature maps of the current frame and historical frames at each level, bilinear interpolation sampling is performed with the coordinate as the center to obtain the multi-scale feature vector at the corresponding position; the sampled features at different levels are then concatenated and used as the image input features for lane decoding.

[0121] Basic model configuration: Input image resolution is Historical window length Preferably, 2 to 5 frames are used. The backbone network can be ResNet-18, MobileNetV3, or other lightweight convolutional networks, and the multi-scale feature resolution can be [missing information - likely a percentage of the input image]. , and ,through Convolution is unified to the number of channels Preferably, The number of lane queries ranges from 128 to 256. The number of rough lane reference points ranges from 16 to 32. The number of fine points ranges from 16 to 32. The range is 32 to 72.

[0122] S2: Perform coarse lane prediction and scene attribute prediction in parallel on the multi-scale image features of the current frame, and output candidate coarse 3D lanes and multi-attribute scene vectors respectively.

[0123] Coarse lane prediction and scene attribute prediction are performed in parallel on the multi-scale image features of the current frame. Specifically, a fixed number of lane queries are used to output candidate coarse 3D lanes for the current frame. Each candidate coarse 3D lane includes at least lane existence, coarse lane reference point 3D coordinates, reference point visibility, lane semantic features, query embedding, and coordinate regression uncertainty. Multi-attribute scene vectors of the current road are output using a multi-label scene attribute branch.

[0124] Each candidate coarse 3D lane includes A coarse lane reference point is sorted according to normalized longitudinal parameters; the coordinate prediction branch outputs the mean and logarithmic variance of the three-dimensional coordinates for each coarse lane reference point. The uncertainty of coordinate regression is determined by the logarithmic variance of the three-dimensional coordinates after exponential mapping; the multi-attribute scene vector includes at least two or more of the following attributes: road straightness attribute, road curvature attribute, road slope attribute, intersection or merging / diverging attribute, tunnel attribute, low illumination attribute, and glare attribute. The probability of each attribute is generated independently by the Sigmoid function.

[0125] The relationship between lane lookup and coordinate prediction branches: Lane lookup is the core component of 3D lane decoding, with each lane corresponding to a query vector. The first branch is used to characterize the lane's identity, semantics, and global geometric attributes. The second branch, which is attached to the lane query decoder, takes the lane query as its core input and combines it with sampled image features to output the coordinates, visibility, and uncertainty of all reference points corresponding to that lane. The two have a "carrier-output branch" relationship: one lane query corresponds to a set of coordinate prediction outputs, and all lane queries share the weights of the coordinate prediction branch.

[0126] A fixed number of lane queries are set on the multi-scale image features of the current frame. Each lane query outputs the lane existence. , The parameters include: ordered candidate coarse lane reference points, reference point visibility, lane semantic probability, query embedding, and coordinate distribution.

[0127] ;

[0128] in, For the first Candidate coarse three-dimensional lane number The mean three-dimensional coordinates of the candidate rough lane reference points; The prediction variance is the corresponding coordinate. The input feature vector for the coordinate prediction branch; It is a coordinate prediction head network, consisting of two layers of 1×1 convolutions and ReLU activation functions, which outputs the mean and logarithmic variance of three-dimensional coordinates; For the first The query embedding vector for each lane; For the current frame image features at the projection position Bilinear sampling characteristics at the location; It is a sinusoidal position encoding function that injects position priors into pixel coordinates.

[0129] The uncertainty of coordinate regression is determined by taking the mean of the logarithmic variance of the three-dimensional coordinates after exponential mapping:

[0130]

[0131] in, For the first The first reference point The prediction variance of the 3D coordinates; This represents the uncertainty in coordinate regression, used to quantify the confidence level of the model's prediction of the 3D coordinates of each reference point. A larger value indicates lower prediction confidence. This uncertainty will subsequently participate in the calculation of historical support confidence, non-uniform sampling density modulation, and geometric prior weight calculation, achieving an adaptive mechanism of "high sampling in low-confidence regions and low constraints in high-confidence regions."

[0132] Coarse lane reference points are sorted according to a unified normalized longitudinal parameter (vehicle direction of travel) and are not mixed with the mathematical control points of the curve basis function. The mathematical control points of the curve basis function are control vertices used to construct spline / Bézier curves, and their positions do not fall directly on the curve, while the coarse lane reference points are sampling points that fall directly on the lane curve. The two have different physical meanings.

[0133] Multi-label scene attribute branching structure and implementation: The multi-label scene attribute branch runs in parallel with the coarse lane branch, and generates scene vectors before temporal fusion, adaptive sampling, and local curve gating to ensure the validity of the data stream. The branching structure is as follows:

[0134] Input: Deepest image features in the current frame ;

[0135] Global average pooling: Feature compression to 3D global feature vector;

[0136] Fully connected hidden layer: The dimensional vector is mapped to a 256-dimensional hidden feature, followed by ReLU activation;

[0137] Output layer: The fully connected layer outputs 7-dimensional raw logits;

[0138] Activation layer: Apply the Sigmoid function independently to each dimension, and output the attribute probability between 0 and 1.

[0139] The corresponding formula is: ;

[0140] in, The original logits vector output for the attribute branches of a multi-label scene; A head network for classifying scene attributes; The deepest image feature in the current frame; For global average pooling; This is the multi-attribute scene vector for the current frame.

[0141] Scene attributes include at least road straightness, road curvature, road slope, intersections or merging / diverging, tunnels, low illumination, and glare. Each attribute is predicted independently and can be simultaneously assigned a high probability. Mutually exclusive softmax classification is not used, making it suitable for real-world road scenes with multiple attributes superimposed.

[0142] The coarse lane decoder consists of "lane query initialization + sparse image feature sampling + coordinate / visibility / semantic / uncertainty prediction head". It sets fixed 3D reference anchor points in the vehicle coordinate system and sparsely samples from the image through camera projection, outputting coarse lane coordinates, reference point visibility, lane semantics, query embedding, and log-variance. The scene attribute branch simultaneously outputs a multi-label scene vector from the deep features of the current frame. Since the scene vector is generated in parallel with the coarse lane, subsequent temporal confidence, sampling coefficients, and local control can all legally utilize the scene vector in the data stream.

[0143] S3: Based on the multi-scale image features of the current frame and the multi-scale image features of the historical frames, predict the relative pose between frames, perform a three-dimensional transformation on the historical candidate coarse lane reference points, and output the transformed historical candidate coarse three-dimensional lanes.

[0144] like Figure 2 As shown, the multi-scale image features of the current frame and the multi-scale image features of the historical frames are input into the inter-frame relative pose prediction branch: when an initial inter-frame pose exists, the six-DOF pose residual relative to the initial inter-frame pose is predicted; when no initial inter-frame pose exists, the complete six-DOF relative pose is predicted; a rigid pose transformation from the historical frame to the current frame is generated by an exponential mapping from Lie algebra to a special Euclidean group, and the rigid pose transformation is used to perform a three-dimensional coordinate transformation only on the historical coarse lane reference point; based on the current coarse lane reference point, the rigid pose transformation, the intrinsic parameter matrix, and the nominally installed extrinsic parameters, sparse projection sampling is performed on the multi-scale image features of the historical frames.

[0145] The specific structure of the pose prediction branch is as follows: The pose prediction branch is a lightweight global regression network that receives compressed global features from the deepest features of the current frame and historical frames. The structure is as follows:

[0146] Input: Global pooling vectors of the deepest features in the current frame and historical frames, along with their element-wise difference and element-wise product, concatenated into a 4×... 3D input vector;

[0147] Hidden layer: 2 fully connected network layers, the first layer has a dimension of 512 and the second layer has a dimension of 256, both followed by ReLU activation and Dropout (dropout rate 0.2).

[0148] Output layer: The fully connected layer outputs a 6-dimensional vector, corresponding to the se(3) Lie algebra parameters (first 3-dimensional translation, last 3-dimensional rotation);

[0149] In residual mode, the output layer weights are initialized to zero, and the biases are initialized to zero to ensure that the pose residual is 0 in the initial stage of training and to match the initial pose prior.

[0150] Two pose prediction modes:

[0151] If a vehicle odometer, low-precision visual odometer, or other initial inter-frame pose exists... Using the right-perturbation residual mode, the output is a right-multiplied six-dimensional pose residual vector. Six-DOF pose residuals: ;

[0152] If no initial inter-frame pose exists, the complete relative pose mode is used to directly predict the complete six-DOF relative pose: ;

[0153] in, For a six-dimensional complete relative pose vector, the symbol is... This represents mapping a six-dimensional vector to a Lie algebra. matrix, Indicates from arrive The exponential mapping no longer mixes in the concept of "residual". Both models use exponential mapping to ensure that the results belong to... .

[0154] SE(3) and Lie algebras: Special Euclidean group It is a Lie group consisting of all rigid transformations (rotation and translation) in three-dimensional space, whose elements are 4×4 homogeneous transformation matrices. ,in It is an orthogonal rotation matrix. It is a translation vector; rigid transformation ensures that the distance and angle of the midpoint in space remain unchanged, which conforms to the physical laws of camera motion. yes The corresponding Lie algebra is a six-dimensional vector space, which can transform constrained pose optimization into unconstrained vector optimization.

[0155] The specific process of exponential mapping is as follows: the six-dimensional vector is split into translation parts. and rotating parts For the rotation vector Calculate the rotation matrix using Rodriguez's formula. ;calculate Left Jacobian matrix The translation term is obtained. The combination forms a homogeneous transformation matrix. This ensures that the results strictly satisfy the rigid transformation constraints.

[0156] The purpose of rigid pose transformation is to unify the 3D lane points in the historical frame coordinate system to the vehicle coordinate system in the current frame, eliminating spatial misalignment caused by vehicle motion and providing a unified coordinate reference for cross-frame association and temporal fusion. Rigid pose transformation only applies to 3D coordinates and does not affect reference point visibility, lane semantic probability, or query embedding. These non-geometric features are transferred based on the association matrix after cross-frame association.

[0157] Historical 3D point coordinate transformation: Historical rough lane reference points are mapped to the current vehicle coordinate system through rigid pose transformation: ;in, For the first time in history Frame number The original coordinates of the coarse reference point for each lane (historical frame vehicle coordinate system); These are the historical coarse reference point coordinates mapped to the vehicle coordinate system of the current frame after rigid pose transformation.

[0158] Historical image sparse projection sampling: To align historical image features, the current coarse lane reference point is used as a sparse geometric reference to transform and project the current point to the historical camera coordinate system. ;in, For the current frame number Lane number The pixel coordinates of each reference point projected onto the historical frame image; This is the camera intrinsic parameter matrix; This is the nominal extrinsic parameter from the vehicle coordinate system to the camera coordinate system; For the current frame number Lane number Coordinates of a coarse reference point; For the inverse pose transformation from the current frame vehicle coordinate system to the historical frame vehicle coordinate system, satisfying .

[0159] Only sampling points with positive depth in the historical camera coordinate system and whose projection positions are located within the image boundary are retained to form a projection validity mask. Subsequently, bilinear sampling is performed on the historical multi-scale features to obtain historical frame multi-scale sampled feature vectors that correspond one-to-one with the current coarse lane reference point.

[0160] The bilinear feature sampling process is as follows: normalize the pixel coordinates of the historical frame obtained by projection to the interval [-1,1], and match the feature map sampling coordinate system; on the feature maps of each level of the historical frame, find the four adjacent pixels according to the normalized coordinates; calculate the bilinear weight according to the pixel distance, and sum the four feature vectors by weight to obtain the sampling feature at that position.

[0161] The historical frame multi-scale image features are not subjected to full-frame dense perspective transformation, but only multi-scale sparse projection sampling is performed at the position corresponding to the current coarse lane reference point to reduce the amount of computation.

[0162] Alignment confidence: Determined jointly by projection effectiveness, local feature similarity, and pose prediction confidence, it measures the reliability of alignment between historical information and the current frame. The calculation formula is:

[0163] ;

[0164] in, To align confidence levels, , These are the feature similarity weight and pose confidence weight, respectively. Due to pose uncertainty, For the current frame image features, Features of historical frame images This is the feature similarity function. The lower the confidence level, the smaller the weight of the temporal fusion, to avoid misaligned information contaminating the current detection result.

[0165] S4: Construct a cross-frame association matrix based on the current candidate coarse 3D lanes and the transformed historical candidate coarse 3D lanes. Based on the cross-frame association matrix, pass the reference point visibility, lane semantic probability and query embedding in the historical candidate coarse 3D lanes to the current frame.

[0166] like Figure 3 As shown, a cross-frame association cost is constructed based on the ordered curve distance, semantic distance, and query embedding distance between the current candidate coarse 3D lane and the transformed historical candidate coarse 3D lane, generating an association matrix from the historical lane to the current lane; the visibility, semantic features, and query embedding of the historical lane are transmitted based on the association matrix, wherein the visibility, semantic features, and query embedding are not transformed through rigid pose transformation.

[0167] Pose transformation can only align positions in 3D space and cannot automatically guarantee that the query sequence number of the historical frame is consistent with the query sequence number of the current frame. Therefore, a cross-frame association cost is constructed before fusion:

[0168] ;

[0169] in, The candidate lane number for the current frame; The candidate lane number in the historical frame; To ensure ordered curve distances calculated using uniform longitudinal parameters, and to avoid direction reversal caused by unordered point set distances; For semantic probability distance, the symmetric Kullback-Leibler distance can be used; To query the embedding distance, 1-cosine similarity can be used; These are the weight coefficients for geometric distance, semantic distance, and embedding distance, respectively, with preferred values ​​of 0.6, 0.2, and 0.2. For the current number Lane and history The total cost of cross-frame association for lanes; the smaller the value, the higher the matching probability. For the current number The semantic probability vector of each lane; For the first time in history The semantic probability vector of each lane; For the current number A set of coarse reference points for each lane; For the history of transformation to the current coordinate system A set of coarse reference points for each lane; For the current number The query embedding vector for each lane; For the first time in history The query embedding vector for each lane.

[0170] The ordered curve distance in the cross-frame association cost is calculated at the location of the common normalized longitudinal parameter. For cases where the historical lane visibility is insufficient, the distance is calculated only at parameter locations that are valid in both the current and historical contexts, and normalized using the number of valid points.

[0171] Methods for generating the correlation matrix:

[0172] Training phase: Sinkhorn normalization is used to form a soft correlation matrix, enabling end-to-end optimization of the correlation module.

[0173] ;

[0174] in, This is the complete cross-frame association cost matrix; The temperature coefficient for Sinkhorn normalization controls the degree of softness or hardness of the distribution; the smaller the value, the sharper the distribution.

[0175] Inference phase: Use Hungarian matching to generate discrete matching relationships for the cross-frame association cost matrix, or perform maximum confidence discretization on the configurable association matrix; invalidate matching pairs with costs greater than the threshold.

[0176] The maximum confidence discretization process is as follows: for each row of the soft correlation matrix (current lane), the historical lane corresponding to the maximum value is selected as the matching result; if the maximum value is lower than the preset threshold (such as 0.3), it is determined that there is no match, and the lane does not integrate historical information.

[0177] Historical feature aggregation is achieved by aggregating historical reference point visibility, lane semantic probability, and query embedding to the current lane identity based on a soft association matrix. Let the soft association matrix be... The historical lane feature matrix is The historical features aggregated to the current lane are: Visibility, semantic features, and query embedding are all transmitted after identity alignment through matrix multiplication, ensuring that historical information corresponds one-to-one with the current lane identity.

[0178] For intersection or merging / diverging scenarios, a current candidate is allowed to have non-zero soft correlations with multiple historical candidates, but the final discrete output is filtered based on the topology prediction results.

[0179] S5: Within the causal time-series window, correct the associated historical frame lane features using the current frame lane features; generate historical frame time-series weights based on historical support confidence and time-series attenuation coefficients; read the corrected historical frame lane features using the current frame lane query based on the historical frame time-series weights, and output the fused lane query through gated residual fusion.

[0180] Within the causal temporal window of future frames without reading the current frame, the lane features of the historical frames are first corrected using the lane features of the current frame, and then the lane features of the historical frames are read using the lane query of the current frame. Based on the temporal interval between the historical frames and the current frame, projection validity, pose alignment confidence, lane visibility, and multi-attribute scene vectors, normalized temporal weights for historical frames are generated to obtain the fused lane query.

[0181] The term "bidirectional" in this invention refers to the existence of two interaction directions between current and historical information, and does not imply reading future frames. The interaction is executed sequentially: first, the current observation is used to correct historical features, and then the current query reads the corrected historical features. The model as a whole is divided into two interaction layers: the image feature layer, which performs temporal interaction at the pixel-level feature dimension, outputting enhanced image features to provide key-value input for the lane query layer; and the lane query layer, which performs temporal interaction at the lane instance dimension, outputting a fused lane query vector for subsequent fine-grained lane decoding.

[0182] Step 1: Correcting historical characteristics based on current observations

[0183] Taking the lane query layer as an example, the first The query for each historical frame, corrected for current observation, is: Let Let the cross-attention output be obtained by using the associated historical features as the query and the current frame features as the key and value.

[0184] ;

[0185] in, The original historical lane features after association and alignment; The cross-attention output obtained by reading the current feature from the historical feature is used to correct the error of the historical feature; This represents the historical lane characteristics after correction based on current observations.

[0186] Historical support confidence calculation: Historical support confidence is jointly determined by pose alignment confidence, projection validity, reference point visibility, local feature similarity, and scene attributes.

[0187] ;

[0188] in, For the first The first historical frame, the first Candidate coarse three-dimensional lanes, the first The confidence level of each reference point is used to calculate the input vector; The pose alignment confidence is obtained by mapping pose prediction uncertainty. This represents the local similarity of features at the same position between the current frame and historical frames. The visibility of the corresponding point in the historical frame; A multi-attribute scene vector; For confidence prediction network (2-layer perceptron); The historical support confidence level ranges from 0 to 1, with higher values ​​indicating more reliable historical information.

[0189] Temporal weight normalization: Historical frame normalization of temporal weights only applies to Historical frame calculation does not include the current frame repeatedly in the historical aggregation:

[0190] ;

[0191] in, For projection and correlation validity mask; The historical support confidence is determined by pose alignment confidence, lane visibility, local feature similarity, and multi-attribute scene vectors. A positive timing decay coefficient; This is the time-series weight normalization factor; Let be the local minimum constant, and let its value be . To prevent division by zero errors; The final normalized temporal weights are calculated by summing the weights of all historical frames to 1. When all historical frames are invalid, the historical aggregation result is set to zero, and the network degenerates into detecting the current frame.

[0192] Step 2: Retrieve historical features for current query

[0193] After retrieving historical features from the current lane query, we obtain: Let This indicates that the query is based on the current lane and the first lane. The cross-attention output obtained by taking each corrected historical feature as a key and a value is as follows:

[0194] ;

[0195] in, This is the original lane query vector for the current frame; The cross-attention output reads the corrected historical features for the current query; This is a lane query that overlays historical information.

[0196] The fused query is then obtained using the gated residual: the gated network generates a gated vector based on the current query and historical augmented queries. The fused query is then:

[0197] ;

[0198] in, For fusion query; This involves element-wise multiplication. The gated residuals are generated by a gated network, and the sigmoid function outputs a gate value between 0 and 1 to achieve adaptive weighted fusion of current and historical information.

[0199] When all historical frames are invalid, the historical aggregation item is zero, and the fusion query degenerates into a query for the current frame. The image feature layer uses the same sequential interaction and uses the output as the key and value for the lane query layer.

[0200] The causal two-layer interaction is executed in the following order: First, using the correlated historical features as the query and the current frame features as the key and value, the historical features corrected by the current observation are obtained; then, using the current frame features as the query and the corrected historical features as the key and value, the enhanced features of the current frame are obtained according to the normalized temporal weights of the historical frames. The image feature layer first allows the historical sampling features to read the current image reference features to suppress historical exposure changes and projection bias; then, it allows the current image reference features to read the corrected historical features. The lane query layer uses the same data flow. The temporal weights are only normalized for historical frames and do not use the current frame as the reference. Repeatedly add historical data to the aggregation. If a historical frame has no effective projection or low association confidence, the model automatically degenerates into single-frame detection.

[0201] S6: Construct non-uniformly distributed parameter anchor points, interpolate candidate coarse 3D lanes at the parameter anchor points to obtain initial 3D points; generate parameter position codes based on the positions of the parameter anchor points, project the initial 3D points onto the current frame image sampling to obtain image sampling features, and combine the parameter position codes, image sampling features, and fused lane query prediction 3D coordinate residuals to obtain fine curve support points.

[0202] The normalized distance, local curvature, road slope, lane integrity, coordinate regression uncertainty, and historical support reliability along the lane direction are calculated based on the candidate coarse 3D lanes. Scene modulation sampling coefficients are generated based on the multi-attribute scene vector, lane integrity, and coordinate regression uncertainty. A fixed number of parameter anchor points with non-uniformly distributed positions are generated using the scene modulation sampling coefficients. Initial 3D points are obtained by interpolating the coarse 3D lanes at the parameter anchor points. The 3D coordinate residuals are predicted based on the fused lane query, parameter position encoding, and corresponding image sampling features. The initial 3D points and the 3D coordinate residuals are added to obtain the fine curve support points.

[0203] This invention first uses coarse lane calculations to determine geometry and confidence levels, then generates parameter anchor points, avoiding the use of undetected fine lanes as sampling criteria. The coarse lanes utilize a unified normalized arc length parameter. The following is represented as .

[0204] Local geometry and confidence calculation:

[0205] The formula for calculating local curvature is: ;

[0206] in, For the rough lane curve with respect to arc length parameter The first derivative (tangent vector); For the rough lane curve with respect to arc length parameter The second derivative; The numerical stability constant for curvature calculations is used to prevent the denominator from being zero, and its value is determined accordingly. In the discrete implementation, the first and second derivatives are approximated by finite difference from the ordered coarse lane reference points.

[0207] The definitions of each geometric and confidence metric are as follows:

[0208] Normalized distance: obtained by normalizing the forward direction coordinates of the coarse reference point, with 0 for the near end and 1 for the far end;

[0209] Local curvature: Calculated by finite difference; the greater the curvature, the sharper the curve.

[0210] Road gradient: calculated by the arctangent of the difference in vertical coordinates between adjacent points and the difference in coordinates in the direction of travel;

[0211] Lane integrity: determined by the mean visibility, the proportion of the largest continuous visible segment, and the image lane response;

[0212] Historical support reliability: Determined by the number of effectively associated historical frames, temporal confidence, and historical trajectory variance, it measures the stability of historical information at this location.

[0213] All physical quantities are normalized before entering the sampling network. The local state vector consists of the normalized distance, normalized curvature, normalized slope, visibility, and coordinate regression uncertainty gated by historical support reliability for each lane location:

[0214] ;

[0215] in, Normalized vertical distance; To normalize the local curvature; To normalize the road slope angle; Visibility probability; Uncertainty in coordinate regression; To ensure historical reliability, high uncertainty is only increased when historical trajectories or current images still provide reliable support, thus avoiding blindly increasing sampling density in completely erroneous or unobserved areas.

[0216] Scene modulation sampling coefficients: Non-negative scene modulation sampling coefficients are generated based on multi-attribute scene vectors, lane integrity, and average coordinate regression uncertainty. The importance of sampling is determined according to the following formula. :

[0217] ;

[0218] in, This is a sampling coefficient prediction network that outputs a modulation coefficient vector. A multi-attribute scene vector; The lane integrity of the current frame; The regression uncertainty of the lane average coordinates in the current frame; This is the bias term for sampling importance, and is a learnable scalar.

[0219] Scene modulation sampling coefficient This allows for increased curvature and slope weights in curve and ramp attributes, improved use of historically reliable features in low-light or glare attributes, and reduced sampling bias related to fixed width in intersection or merging / diverging attributes. All coefficients are kept non-negative using softplus.

[0220] Non-uniform parameter anchor point generation: for sampling importance Perform normalization and construct the cumulative distribution:

[0221] ;

[0222] in, It is a very small constant to ensure the minimum sampling probability; For the first The importance of sampling at each point; This represents the cumulative distribution function value.

[0223] A fixed number of parameters are generated within the unified normalized arc length parameter domain through cumulative distribution inversion. The parameter anchor points, specifically the process, are as follows: M quantiles are uniformly generated in the interval [0,1], with the first and last points fixed at 0 and 1 respectively; for each quantile, the cumulative distribution is retrieved by linear interpolation. The corresponding arc length parameter is obtained. These are non-uniform parameter anchor points; a minimum spacing constraint is applied between adjacent anchor points (not less than 1 / M of the total arc length) to prevent excessive clustering of anchor points. This generates parameter positions, not 3D curve control points.

[0224] The mapping from parameter anchor points to fine curve support points is as follows:

[0225] At each parameter anchor point, ordered interpolation of the coarse 3D lane is performed to obtain the initial 3D points:

[0226] ;

[0227] in, These are the initial three-dimensional points; For the rough lane reference point set; For the first Anchor points for each parameter; For ordered interpolation on coarse three-dimensional lanes, piecewise linear interpolation, cubic interpolation, or differentiable spline interpolation can be used.

[0228] The current frame image features obtained by initial 3D point projection sampling Combined with integrated query and parameter position encoding Predicting 3D coordinate residuals:

[0229] ;

[0230] in, These are the predicted values ​​of the three-dimensional coordinate residuals; The residual prediction head network consists of two layers of 1×1 convolutions plus ReLU; These are fine-grained curve support points used for curve reconstruction. This mapping explicitly completes the technical chain of "non-uniform parameter position - three-dimensional curve support point".

[0231] S7: Construct multiple piecewise curve basis functions using fine curve support points, and then weight and fuse the outputs of the multiple piecewise curve basis functions to output fine three-dimensional lane coordinates in the vehicle coordinate system.

[0232] like Figure 4As shown, multiple piecewise curve basis functions are constructed using fine curve support points. Local soft-gating weights are generated point-by-point or segment-by-segment based on fused lane query, parameter position encoding, local visibility, local coordinate regression uncertainty, and multi-attribute scene vectors. The 3D outputs of multiple piecewise curve basis functions at the position of the unified normalized arc length parameter are weighted and fused. Simultaneously, the visibility of fine lane points is predicted, and the fine 3D lane coordinates, fine lane point visibility, lane existence, lane semantic category, and corresponding confidence score in the vehicle coordinate system are output.

[0233] Multiple piecewise curve basis functions: The preferred multiple piecewise curve basis functions include piecewise Catmull-Rom interpolation curves, cubic B-spline curves, and piecewise cubic Bézier curves. All three branches use the same normalized arc length parameter.

[0234] The piecewise cubic Bézier curve The segment is represented as:

[0235] ;

[0236] in, For the first Output the coordinates of a cubic Bézier curve; Intra-segment parameters of Power of; For the first The first segment of the Bezier curve One control point.

[0237] Apply continuity constraints to adjacent segments:

[0238] ;

[0239] This constraint means that the endpoint of the previous Bézier curve coincides with the starting point of the next segment (C0 position is continuous), and the tangent vector at the endpoint is equal to the tangent vector at the starting point (C1 first derivative is continuous), ensuring that the entire piecewise Bézier curve transitions smoothly without abrupt inflection points, conforming to the physical shape of the lane line.

[0240] The three types of basis functions are defined as follows: the Catmull-Rom branch is used to maintain the local shape near the observation support point; the cubic B-spline branch is used to reduce local noise and maintain the second-order smooth trend; and the piecewise cubic Bessel branch is used to continuously complete long occluded sections.

[0241] Local soft gating weights: Local gating weights are generated based on parameter location or local segments.

[0242] ;

[0243] in, The input feature vector for local control; For local visibility; This represents uncertainty in local coordinate regression. A multi-attribute scene vector; For gated prediction networks; For the first A basis function gated weight vector at each position.

[0244] Local control network structure: The input is a feature vector of fusion query, location encoding, local visibility, uncertainty, and scene vector concatenation; it is mapped to a 3D weight vector through 2 fully connected layers, and finally normalized by Softmax to output the weights of three curve basis functions, with the weights summing to 1.

[0245] The final refined 3D lane points are:

[0246] ;

[0247] in, This is the set of curve basis functions. Point-by-point or segment-by-segment gating allows different basis function preferences to be used for the near-end intact segment, the middle noisy segment, and the far-end occluded segment of a lane, instead of describing the entire lane with a single global weight.

[0248] The technical advantages of local multi-basis function soft gating:

[0249] Clear near-end sections: Increase the weight of Catmull-Rom branches to accurately preserve local lane line details and wear characteristics;

[0250] In the middle noise section: increase the weight of the B-spline branch to suppress curve jitter caused by image noise;

[0251] Far-end occlusion section: Increase the weight of segmented Bezier branches and use curve continuity to achieve smooth completion over long distances; compared with a single global weight, local control can simultaneously take into account local detail fidelity, overall smoothness and occlusion completion capability, and adapt to the differences in observation conditions of different lane lines.

[0252] Fine-grained visibility prediction: Fine-grained 3D lane point visibility is re-predicted by an independent fine-grained visibility head, without directly interpolating the coarse reference point visibility.

[0253] ;

[0254] in, The input feature vector provides fine-grained visibility; For fine-grained visibility prediction networks; For the first The visibility probability of each fine point. Therefore, even the number of fine points... Number of coarse reference points Although they differ, the final three-dimensional coordinates and visibility still have the same dimensions.

[0255] To mitigate the risk of gating degenerating into a single branch during training, load-balanced loss can be used at the batch level, without forcing each sample to have high entropy, thus preserving the specialization of each branch for specific scenarios. The final output includes a fixed number of fine points and their visibility. Post-processing generates variable-length effective lanes based on lane existence, fine visibility, and detection range.

[0256] Multi-attribute scene modulation and bounded geometric priors:

[0257] Multi-attribute scene vectors are generated during the joint coarse prediction stage of the current frame and then input into the sampling coefficient network, historical support confidence network, local control network, and fine decoder, respectively. Therefore, the generation order of scene attributes precedes the order in which they are invoked by the modules.

[0258] During the training phase, bounded geometric prior weights are generated based on the scene vector, lane completeness, and average uncertainty.

[0259] make and order ,but: ,in, , The input vector for calculating geometric prior weights. For multi-attribute scene vectors, For lane integrity, For the uncertainty of mean coordinate regression, For a priori weight prediction network, These are the normalized weighting coefficients. This is a positive lower bound for the geometric prior weights, ensuring that the prior does not become completely invalid. These are geometric prior weights.

[0260] Geometric priors include at least smoothness priors, adjacent lane direction consistency priors, adjacent lane width priors, and topological priors, with corresponding loss calculations as follows:

[0261] Smoothing Prior Loss: Calculate the second-order difference smoothing loss for fine points:

[0262] make ,but:

[0263] ;

[0264] in, For the first Lane number Second-order difference smoothing error of points; For the first Lane 1 The three-dimensional coordinates of each parameter position; The sum of effective point errors; As a visibility effectiveness mask, only visible points participate in the loss calculation; The number of valid points; For fine-point smoothing loss.

[0265] Adjacent lane direction consistency prior loss: Define the lane tangent vector at the location of common parameters, and calculate the direction difference between adjacent lanes: Let ,but:

[0266] ;

[0267] in, Adjacent lanes and In the Cosine similarity of the tangent vectors of the points; This represents the directional difference value. This is the corresponding lane tangent vector; As a topology-effective mask, the intersection / merging / diverging region is set to zero; This refers to the loss of directional consistency between adjacent lanes.

[0268] Adjacent lane width loss:

[0269] make ,but:

[0270] ;

[0271] in, This represents the prediction error for the width of adjacent lanes; This represents the predicted lateral distance between adjacent lanes. The preset standard lane width is the a priori value; This is the loss of width between adjacent lanes.

[0272] Adjacent relationships are established by sorting visible lanes by lateral coordinates at common longitudinal locations; when intersection or merging / diverting attributes exceed a threshold, the topology validity mask closes inapplicable directional consistency and fixed width constraints to avoid excessive suppression of the real road structure by prior knowledge.

[0273] The total geometric prior loss is:

[0274] ;

[0275] in, , , , These are the weighting coefficients for smoothness, directional consistency, width, and topological prior, respectively. The topological prior loss, constraining the changes in the number of lanes and the rationality of merging and diverging locations, consists of lane ordering loss and bifurcation point location loss. For fine-point smoothing loss, This is due to the loss of directional consistency between adjacent lanes. For adjacent lane width loss, scene modulation and bounded prior structure such as Figure 5 As shown.

[0276] Scene attribute labels can be manually labeled from the dataset, or weak labels can be generated from 3D lane curvature, slope, topological labels, average image brightness, and highlight ratio. Scene branches use multi-label binary cross-entropy, not mutually exclusive softmax.

[0277] Example 2

[0278] This embodiment proposes a lightweight 3D lane training method with scene-adaptive spatiotemporal coordination to optimize the detection network described in Embodiment 1. The training method and the online detection method are defined separately to avoid the training-inference inconsistency problem.

[0279] The steps include: constructing a bipartite graph matching cost between the predicted lane and the real lane in the current frame to complete lane-level matching; calculating lane existence loss, coarse lane and fine lane 3D coordinate loss, ordered curve loss, fine lane point visibility loss, lane semantic loss, and negative log-likelihood loss of coordinate uncertainty based on the matching results; calculating pose supervision loss based on SE(3) logarithmic mapping when the training data provides real inter-frame pose; calculating projection feature consistency loss and inter-frame cycle consistency loss when the training data does not provide real inter-frame pose; calculating inter-frame temporal consistency loss at positions where cross-frame association is effective, projection is effective, and visibility is effective; generating geometric prior weights with positive lower bounds and preset upper bounds based on multi-attribute scene vectors, lane completeness, and coordinate regression uncertainty, and calculating smoothing prior loss, adjacent lane direction consistency prior loss, lane width prior loss, and lane topology prior loss under topological validity mask constraints; weighted summing of the above detection loss, attribute loss, temporal loss, and regularization loss to obtain the total loss, and jointly optimizing the overall parameters of the 3D lane detection network through backpropagation.

[0280] The core training process is as follows:

[0281] 1. Bipartite Graph Matching: During the training phase, a bipartite graph matching cost is first constructed for the candidate coarse lanes and the real lanes in the current frame. The matching cost includes at least the existence cost, the ordered 3D distance of the coarse lanes, the ordered distance of the fine curves, and the semantic cost. Based on the matching results, a one-to-one correspondence between the predicted lanes and the real lanes is established.

[0282] 2. Detection and Attribute Loss Calculation:

[0283] Lane existence loss: binary cross-entropy loss;

[0284] 3D coordinate loss for coarse and fine lanes: Smooth L1 loss;

[0285] Order curve loss: shape consistency loss along the normalized arc length parameter;

[0286] Fine-grained lane point visibility loss: binary cross-entropy loss;

[0287] Lane semantic loss: multi-class cross-entropy loss;

[0288] Coordinate uncertainty loss: heteroscedasticity negative log-likelihood loss.

[0289] The formula for calculating the negative log-likelihood loss due to coordinate uncertainty is:

[0290] make ,but:

[0291] ;

[0292] in, The squared error of single-point, one-dimensional coordinate prediction; The coordinates are the actual labeled values; This represents a single-point, single-dimensional loss value. This represents the total uncertainty loss.

[0293] Pose branch loss calculation:

[0294] When the training data provides the true inter-frame poses, calculate the pose supervision loss based on the SE(3) logarithmic mapping:

[0295] ;

[0296] in, For real-time inter-frame pose annotation; The SE(3) logarithmic mapping transforms the transformation matrix into a six-dimensional Lie algebra vector.

[0297] When the training data does not provide the true inter-frame pose, calculate the projection feature consistency loss and the inter-frame cycle consistency loss:

[0298] ;

[0299] ;

[0300] in, This refers to the alignment error of the projected features. These are the current and historical sampling features at the same location, respectively. For projection feature consistency loss; To achieve cyclic consistency loss, the reciprocity of forward and reverse pose transformations is constrained.

[0301] Temporal consistency loss: At positions where cross-frame association, projection, and visibility are effective, calculate the temporal consistency loss of lane coordinates between frames to constrain the stability of detection results in consecutive frames.

[0302] Geometric prior loss: Generate geometric prior weights with positive lower bounds and preset upper bounds based on multi-attribute scene vectors, lane integrity, and coordinate regression uncertainty, and calculate smoothness, adjacent lane direction consistency, lane width, and lane topology prior loss within the topology validity mask.

[0303] Total Loss and Joint Optimization: The total loss is a weighted sum of the detection loss group, attribute loss group, temporal loss group, and regularization loss group.

[0304] Loss detection group: ;

[0305] Attribute loss group: ;

[0306] Time-series loss group: ;

[0307] Regularized loss group: ;

[0308] Total loss: ;

[0309] Among them, each These are the weighting coefficients corresponding to the loss; The loss method can be either pose loss or projection feature consistency loss. The choice between supervised loss and a combination of alignment and recurrence loss can be made based on the training data conditions. Constrain cross-frame association matching, This is a batch-level gating load balancing loss mechanism used to reduce the risk of multi-curve branches degrading into a single branch over a long period. For lane existence loss, For the loss of the three-dimensional coordinates of the rough lane, For the loss of fine lane three-dimensional coordinates, For ordered curve loss, For fine lane point visibility loss, For lane semantic loss, The negative log-likelihood loss is due to coordinate uncertainty. For time-series consistency loss, For scene modulation sampling coefficient loss, The loss is a geometric prior. Each loss is calculated only within the corresponding existence, visibility, projection validity, association validity, and topological validity mask.

[0310] Phased training strategy:

[0311] Training is divided into three stages to gradually improve model stability:

[0312] Phase 1: Train the coarse lane branch, scene attribute branch, and uncertainty branch of the current frame, freeze the temporal sequence and fine reconstruction modules, and establish basic single-frame detection capabilities.

[0313] The second stage involves adding pose estimation, projection sampling, and cross-frame association modules, using the pre-trained weights from the first stage as initialization, to train temporal alignment and association capabilities.

[0314] Phase 3: Joint training of causal fusion, non-uniform anchor points, local multi-basis functions, and bounded geometric prior modules, followed by end-to-end optimization of the entire network. In the initial stages of joint training, temporal and prior loss weights are gradually increased to reduce the impact of coarse detection instability on temporal branches.

[0315] Reasoning process and lightweight implementation:

[0316] The maintenance length of the inference phase is The system includes multi-scale feature caching, coarse lane caching, and query embedding caching. Upon arrival of a new image, the current frame joint prediction module outputs coarse lanes, uncertainties, and scene vectors; the pose module estimates the transformation from each cached frame to the current frame; after completing sparse projection and cross-frame association, causal two-layer interaction is performed; subsequently, parameter anchor points and fine support points are generated, local multi-basis function reconstruction is performed, and fine lanes are output.

[0317] This invention does not construct a complete dense temporal bird's-eye view, but only around... A rough reference position and Sparse sampling and interaction are performed at precise locations. The main time-series computational complexity is... It is directly proportional to the number of pixels in the entire image, rather than being multiplied directly by the number of pixels in the image. For the length of the history window, For the number of lane queries, For the number of fine points, For feature channel dimensions. The preferred configuration is lane query. 1. Rough lane reference point Fine curve support points Historical window length Number of channels It has good potential for lightweight real-time deployment.

[0318] Example 3

[0319] like Figure 6As shown, based on the method of the above embodiments, this embodiment provides a three-dimensional lane detection system, including an image and camera parameter input module, a current frame joint coarse prediction module, a scene attribute modulation module, an inter-frame pose and sparse projection alignment module, a cross-frame lane association module, a causal two-layer temporal interaction module, an adaptive parameter anchor point and fine support point generation module, a local multi-basis function reconstruction module, a three-dimensional lane output module, and a training constraint module.

[0320] The functions of each module are defined as follows:

[0321] Image and camera parameter input module: Receives time-series road images acquired by the vehicle-mounted monocular camera, reads the camera intrinsic parameter matrix and nominal mounting extrinsic parameter matrix, performs preprocessing such as image size normalization and color standardization, and completes the initial definition of the vehicle coordinate system and camera coordinate system, providing a unified input data benchmark for subsequent modules.

[0322] The current frame joint coarse prediction module extracts multi-scale image features of the current frame through a lightweight backbone network with shared weights. Based on sparse sampling of 3D reference anchor projection, it outputs candidate coarse 3D lanes, coordinate regression uncertainty, lane semantic features and multi-attribute scene vectors in parallel to complete preliminary geometric perception and scene parsing.

[0323] Scene attribute modulation module: Input multi-attribute scene vectors into the sampling coefficient network, historical support confidence network, local control network and geometric prior weight generation unit respectively to realize the adaptive modulation of scene attributes on the whole link, so that the sampling density, temporal weight, basis function preference and prior strength are dynamically adjusted with the road scene.

[0324] Inter-frame pose and sparse projection alignment module: Based on the deep image features of the current frame and the historical frame, predict the rigid relative pose of the inter-frame SE(3), perform coordinate transformation on the historical 3D lane points to achieve spatial geometric alignment; at the same time, back-project the current coarse lane points to the historical frame image, perform multi-scale sparse bilinear feature sampling, and output the aligned historical image features and projection validity mask.

[0325] Cross-frame lane association module: Combines ordered curve geometric distance, semantic probabilistic distance and query embedding distance to construct a multi-dimensional association cost matrix. During the training phase, a soft association matrix is ​​generated by Sinkhorn normalization. During the inference phase, discrete matching relationships are generated by Hungarian matching to establish the identity correspondence between historical lanes and current lanes, thus solving the problem of lane number misalignment between frames.

[0326] Causal two-layer temporal interaction module: Within a causal window containing only current and historical frames, sequential bidirectional interaction of "current correction history - current reading history" is performed in the image feature layer and lane query layer respectively; normalized temporal weights are generated by combining temporal intervals, alignment confidence and scene attributes, and the enhanced fused lane query is output through gated residual fusion.

[0327] The adaptive parameter anchor point and fine support point generation module combines coarse lane geometric attributes, coordinate uncertainty, historical support reliability, and scene attributes to generate non-uniformly distributed parameter anchor points. It obtains initial 3D points through coarse curve interpolation, and combines fusion query and image feature prediction to obtain 3D coordinate residuals, ultimately obtaining fine curve support points, thus realizing the skewed allocation of computing resources to complex sections.

[0328] Local multi-basis function reconstruction module: Based on fine support points, construct segmented Catmull-Rom, cubic B-spline, and segmented cubic Bézier curve basis functions; generate point-by-point or segment-by-segment local soft-gating weights, and weighted fuse the multi-basis function outputs to obtain the final fine three-dimensional lane, and simultaneously predict the visibility of fine points and lane confidence.

[0329] The 3D lane output module performs post-processing on the original detection results, including invalid lane filtering, visible segment extraction, continuous lane stitching and format standardization, and finally outputs structured perception results such as fine 3D lane coordinates, visibility, semantic category, and existence confidence in the vehicle coordinate system.

[0330] Training Constraints Module: Enabled only during model training, it is responsible for constructing all training constraints such as bipartite graph matching, coordinate loss, temporal loss, and geometric prior loss, calculating the total loss and performing backpropagation and parameter updates to achieve end-to-end optimization of the network.

[0331] Example 4

[0332] This embodiment provides an electronic device, including at least one processor, a memory, an image input interface, and a communication interface. The memory stores a computer program, and when the processor executes the computer program, it implements the scene-adaptive spatiotemporal collaborative lightweight 3D lane detection method described in Embodiment 1, or the 3D lane network training method described in Embodiment 2.

[0333] Electronic devices can be in-vehicle computing platforms, autonomous driving domain controllers, edge computing devices, or cloud servers. When deployed in vehicles, they can be adapted to embedded computing platforms to meet the requirements of real-time 3D lane perception; when deployed in the cloud, they can be used for model training and batch data processing of large-scale datasets.

[0334] Example 5

[0335] This embodiment provides a computer-readable storage medium on which a computer program is stored. The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a solid-state drive, flash memory, an optical disk, a magnetic medium, or other media capable of storing computer program instructions.

[0336] When the computer program is executed by the processor, it can implement the scene-adaptive spatiotemporal collaborative lightweight three-dimensional lane detection method described in Embodiment 1 above, or implement the three-dimensional lane network training method described in Embodiment 2.

[0337] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination, characterized by the following steps: include: S1: Extract multi-scale image features of the current frame and multi-scale image features of historical frames through a lightweight backbone network; A three-dimensional reference anchor point is set in the vehicle coordinate system and projected onto the current frame image. Image features are sparsely sampled at the projection position, and the three-dimensional coordinate offset is predicted to obtain candidate coarse lane reference points. S2: Perform coarse lane prediction and scene attribute prediction in parallel on the multi-scale image features of the current frame, and output candidate coarse 3D lanes and multi-attribute scene vectors respectively; S3: Based on the multi-scale image features of the current frame and the multi-scale image features of the historical frames, predict the relative pose between frames, perform a three-dimensional transformation on the historical candidate coarse lane reference points, and output the transformed historical candidate coarse three-dimensional lanes. S4: Construct a cross-frame association matrix based on the current candidate coarse 3D lanes and the transformed historical candidate coarse 3D lanes. Based on the cross-frame association matrix, pass the reference point visibility, lane semantic probability and query embedding in the historical candidate coarse 3D lanes to the current frame. S5: Within the causal time window, correct the associated historical frame lane features using the current frame lane features; Historical frame temporal weights are generated based on historical support confidence and time-series decay coefficients; based on the historical frame temporal weights, the corrected historical frame lane features are read using the current frame lane query, and the fused lane query is output through gated residual fusion. S6: Construct non-uniformly distributed parameter anchor points, interpolate candidate coarse 3D lanes at the parameter anchor points to obtain initial 3D points; generate parameter position codes based on the positions of the parameter anchor points, project the initial 3D points onto the current frame image sampling to obtain image sampling features, combine the parameter position codes, image sampling features and the fused lane query prediction 3D coordinate residuals to obtain fine curve support points; S7: Construct multiple piecewise curve basis functions using fine curve support points, and then weight and fuse the outputs of the multiple piecewise curve basis functions to output fine three-dimensional lane coordinates in the vehicle coordinate system.

2. The lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination according to claim 1, characterized in that, The method for setting three-dimensional reference anchor points in the vehicle coordinate system and projecting them onto the current frame image is as follows: ； in, For the first Candidate coarse three-dimensional lane number The projected pixel coordinates of each 3D reference anchor point 3D reference anchor point homogeneous three-dimensional reference anchor points, For the camera intrinsic parameter matrix, Install the extrinsic parameter matrix in the nominal direction from the camera coordinate system to the vehicle coordinate system. To normalize homogeneous coordinates; The method for sparsely sampling image features at the projection location is as follows: For each projected pixel coordinate Based on multi-scale image features of the current frame and historical frames, using projected pixel coordinates Bilinear interpolation sampling is performed around the center to obtain multi-scale features; The candidate rough lane reference point for: ;in, This represents the three-dimensional coordinate offset.

3. The lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination according to claim 2, characterized in that, The method for performing coarse lane prediction and scene attribute prediction in parallel on the multi-scale image features of the current frame, and outputting candidate coarse 3D lanes and multi-attribute scene vectors respectively, is as follows: Set a fixed number of lane queries on the multi-scale image features of the current frame; predict the lane existence, reference point visibility and lane semantic probability of each lane query based on the multi-scale image features of the current frame. For each lane query, project pixel coordinates Multi-scale features obtained from sampling Combined with sinusoidal position coding and query embedding Constructing input features for the coordinate prediction branch for: ； Input features Input coordinate prediction branch, synchronously output the mean 3D coordinates of each candidate rough lane reference point and the logarithmic variance of three-dimensional coordinates for: ;in, For coordinate prediction head network; Logarithmic variance of three-dimensional coordinates Determine the uncertainty of coordinate regression for: in, For spatial dimension indexing, These represent the three coordinate axes in three-dimensional space. For candidate rough lane reference point number Log-variance on 3D coordinates; Candidate coarse 3D lanes are constructed by considering lane existence, reference point visibility, lane semantic probability, query embedding, 3D coordinate mean, and coordinate regression uncertainty. The deepest image features of the current frame are analyzed through the scene attribute branch. Perform global average pooling and fully connected hidden layer mapping to output multidimensional raw logical values. for: ;in, For the current frame index, For global average pooling; A head network for classifying scene attributes; For multidimensional primitive logical values Applying a Sigmoid activation function to each dimension, the output is a multi-attribute scene vector. for: .

4. The lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination according to claim 3, characterized in that, The method for predicting the inter-frame relative pose based on the multi-scale image features of the current frame and the multi-scale image features of historical frames, performing a 3D transformation on the historical candidate coarse lane reference points, and outputting the transformed historical candidate coarse 3D lanes is as follows: Input the multi-scale image features of the current frame and the multi-scale image features of the historical frames into the pose prediction branch, and output the inter-frame relative pose transformation from the historical frames to the current frame. ; Based on inter-frame relative pose transformation A 3D transformation is performed on the historical candidate coarse lane reference points to obtain the transformed historical candidate coarse lane reference points: ;in, For historical frame indexing, For historical frames Candidate coarse lane reference points for candidate coarse 3D lanes. The transformed historical candidate coarse lane reference point; The current candidate crude lane reference point By mapping the inverse inter-frame relative pose transformation from the current frame to the historical frame onto the historical frame image plane, the projected pixel coordinates of the historical frame are obtained. for: ;in, This represents the inverse inter-frame relative pose transformation from the current frame to historical frames. ; Projecting pixel coordinates in historical frames At this point, bilinear interpolation sampling is performed on the multi-scale image features of historical frames to obtain the multi-scale features of historical frames.

5. A lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination according to claim 4, characterized in that, The method for constructing a cross-frame correlation matrix based on the current candidate coarse 3D lanes and the transformed historical candidate coarse 3D lanes is as follows: Cost of constructing cross-frame associations: ； in, For the current number Candidate coarse three-dimensional lanes and historical first The cross-frame association cost of candidate coarse 3D lanes, For ordered curve distances, For lane semantic probability distance, To query the embedding distance, These are the weight coefficients for ordered curve distance, lane semantic probability distance, and query embedding distance, respectively. For the current number Candidate coarse lane reference points for candidate coarse 3D lanes. For the transformed history Candidate coarse lane reference points for candidate coarse 3D lanes. For the current number The semantic probability of each candidate coarse 3D lane. For the first time in history The semantic probability of each candidate coarse 3D lane. For the current number Query embedding of candidate coarse 3D lanes, For the first time in history Query embedding of candidate coarse 3D lanes; Cross-frame association cost Constructing the cross-frame association cost matrix A cross-frame correlation matrix is ​​generated by Sinkhorn normalization. for: ,in, For temperature coefficient, This is the normalization function for a double random matrix.

6. A lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination according to claim 5, characterized in that, The historical frame lane features are corrected and associated with the current frame lane features; Historical frame temporal weights are generated based on historical support confidence and time-series attenuation coefficients. The method for reading corrected historical frame lane features using the current frame lane query, and then outputting the fused lane query via gated residual fusion, is as follows: Using the current frame lane features as keys and values, and the associated historical frame lane features as queries, cross-attention is performed to obtain the corrected historical frame lane features: ； in, For the lane features of the associated historical frames, This is the cross-attention output obtained by using historical frame lane features as the query and current frame lane features as the key and value. Lane features in the corrected historical frames; Calculate historical support confidence for: ;in, For pose alignment confidence. For local feature similarity, For the visibility of the reference point, For confidence prediction networks; Calculate the normalized historical frame temporal weights for: ； in, For projection and correlation validity mask, A positive time-series decay coefficient. It is the minimum constant. For the first The original temporal weights of each historical frame; Using the current frame lane query as the query and the corrected historical frame lane features as the key and value, cross-attention is performed to obtain the lane query after overlaying historical information. for: ； in, For the current frame lane query, The cross-attention output reads the corrected historical frame lane features for the current frame lane query; Gated residual fusion output fusion lane query for: ； in, For the gated vector, This is for element-wise multiplication.

7. A lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination according to claim 6, characterized in that, The method for constructing non-uniformly distributed parameter anchor points is as follows: Based on the local geometric features and coordinate regression uncertainties along the lane direction calculated from the candidate coarse 3D lanes, a local state vector is formed. for: ;in, To normalize the vertical distance, To normalize the local curvature, To normalize the road slope angle, For the visibility probability of the reference point, For coordinate regression uncertainty, To support historical reliability; Based on multi-attribute scene vectors Lane integrity and the uncertainty of mean coordinate regression Generate modulation sampling coefficients for: ;in, For modulation sampling coefficient prediction network, It is a positive value mapping function; Calculate the importance of sampling for: ;in, This is a bias term for the importance of sampling; Importance of sampling Normalization is performed and a cumulative distribution is constructed. The non-uniformly distributed parameter anchor points are then generated through inversion of the cumulative distribution. ; The initial three-dimensional points for: ;in, For the set of candidate coarse lane reference points, It is an interpolation function; The three-dimensional coordinate residual for: ;in, For residual prediction head network, Encode the parameter position. Image sampling features; The fine curve support point for: .

8. A lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination according to claim 6 or 7, characterized in that, The method for constructing multiple piecewise curve basis functions using fine curve support points, and then weighting and fusing the outputs of these multiple piecewise curve basis functions to output fine three-dimensional lane coordinates in the vehicle coordinate system is as follows: Calculate the local basis function gate weights for: ; in, For local visibility, For local coordinate regression uncertainty, For gated prediction networks; The fine three-dimensional lane coordinates for: ;in, For the set of basis functions of the curve, For the first Each basis function at the parameter anchor point Gating weights at the location, For the first Each basis function at the parameter anchor point The output at that location.

9. A lightweight 3D lane training method with scene-adaptive spatiotemporal coordination, employing the lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination as described in any one of claims 1 to 8, characterized in that, include: Construct a bipartite graph matching cost between the predicted 3D lanes and the actual 3D lanes in the current frame to complete lane-level matching; Based on the matching results, the detection loss, attribute loss, temporal loss and regularization loss are calculated, and the total loss is obtained by weighted summation. Based on the total loss, the coordinate prediction branch parameters, scene attribute branch parameters, pose prediction branch parameters, confidence prediction network parameters, modulation sampling coefficient prediction network parameters, residual prediction head network parameters, and gated prediction network parameters in the 3D lane detection method are jointly optimized through backpropagation.

10. A lightweight 3D lane detection system with scene-adaptive spatiotemporal coordination, employing the lightweight 3D lane detection method with scene-adaptive spatiotemporal coordination as described in any one of claims 1 to 8, characterized in that, include: The image and camera parameter input module is used to receive temporal road images and camera calibration parameters, and perform image preprocessing; The current frame joint coarse prediction module is used to extract multi-scale image features of the current frame, perform coarse lane prediction and scene attribute prediction in parallel, and output candidate coarse 3D lanes and multi-attribute scene vectors. The scene attribute modulation module is used to input multi-attribute scene vectors into each modulation unit to achieve scene adaptive adjustment of sampling density, temporal weight and basis function fusion weight; The inter-frame pose and sparse projection alignment module is used to predict the inter-frame relative pose, perform 3D transformation on historical candidate coarse lane reference points, and perform sparse projection sampling on the multi-scale image features of historical frames. The cross-frame lane association module is used to construct a cross-frame association matrix and establish the correspondence between historical candidate coarse 3D lanes and current candidate coarse 3D lanes. The causal dual-layer temporal interaction module is used to correct the lane features of the historical frame with the lane features of the current frame within the causal temporal window, and to read the corrected lane features of the historical frame with the lane query of the current frame, and output the fused lane query through gated residual fusion. The adaptive parameter anchor point and fine support point generation module is used to construct non-uniformly distributed parameter anchor points, interpolate candidate coarse 3D lanes to obtain initial 3D points, and combine fused lane query, parameter position encoding and image sampling features to predict 3D coordinate residuals to obtain fine curve support points. The local multi-basis function reconstruction module is used to construct multiple piecewise curve basis functions, and to perform weighted fusion of the outputs of multiple piecewise curve basis functions to output fine three-dimensional lane coordinates in the vehicle coordinate system. The 3D lane output module is used to perform post-processing on the detection results and output standardized 3D lane perception results. The training constraint module, enabled only during the training phase, is used to construct the loss function and perform network parameter optimization.

Images