Vehicle-mounted laser radar and visual sensor combined positioning method

Abstract

The application discloses a kind of vehicle-mounted laser radar and visual sensor combined positioning method.In the present application, by introducing cross-modal hallucination field, using historical joint distribution prior, the semantic topological skeleton of current scene and high-confidence heterogenous anchor points, generate hallucination features highly consistent with real modal in static area.In energy optimization, prior maintenance term ensures that hallucination results follow the cross-modal mapping rules in physics, topological control term limits hallucination within stable spatial structure, and anchor alignment term forces local features to maintain geometric semantic consistency with immutable geometric seeds.These triple constraints adjust the spatial anchoring parameters of hallucination field in real time through pose update, so that hallucination features are always accurately aligned with the current vehicle perspective, effectively reducing the risk of positioning jump and inlier collapse, and significantly improving the no-intervention operation capability of intelligent driving under complex lighting and weather conditions.

Description

Technical Field

[0001] This invention belongs to the field of automotive sensor technology, specifically a method for joint positioning using vehicle-mounted lidar and vision sensors. Background Technology

[0002] Multi-sensor fusion positioning for automobiles refers to a cutting-edge intelligent connected vehicle technology that comprehensively processes heterogeneous sensor data from multiple sources, including Global Navigation Satellite System (GNSS), Inertial Measurement Unit (IMU), LiDAR, millimeter-wave radar, and high-definition cameras, and uses advanced algorithms to achieve high-precision global pose estimation of the vehicle. Its core logic lies in using data fusion algorithms such as Kalman filtering and factor graph optimization to overcome the physical limitations of single sensors: GNSS provides global absolute coordinates, IMU compensates for high-frequency dynamic pose, and in scenarios such as urban canyons or tunnels where satellite signals are obstructed, real-time matching and wheel speed calculation are performed using LiDAR point clouds and visual features to maintain centimeter-level continuous positioning. This technology effectively overcomes pain points such as GNSS multipath effects, visual perception being affected by adverse lighting and weather conditions, and radar cumulative drift, achieving highly reliable pose output in all weather and all scenarios. As the underlying foundation for perception and control in advanced autonomous driving systems, multi-sensor fusion positioning not only provides an indispensable spatial reference for vehicle lane-level navigation and trajectory planning, but also serves as a core technological barrier to ensure the safe operation of intelligent vehicles in complex and dynamic traffic environments.

[0003] However, in existing technologies, multi-sensor fusion positioning systems commonly suffer from pose jumps or even positioning loss due to sensor degradation under adverse conditions such as nighttime, rain, and fog. Traditional approaches either discard data from failed modes, resulting in information waste, or rely on pre-calibrated fixed weights for fusion, which cannot adapt to dynamically changing sensor confidence levels. Furthermore, existing methods for relocalization after positioning loss are highly dependent on the sensor observation quality of the current frame; once observations degrade, the relocalization success rate drops sharply. In addition, most systems lack the ability to learn and evolve from degradation events; when faced with repeated failures in the same scenario, they can only compensate by adding redundant sensors or high-precision maps, leading to continuously rising hardware costs and maintenance expenses. Summary of the Invention

[0004] The purpose of this invention is to provide a method for joint positioning of vehicle-mounted LiDAR and visual sensors in order to solve the problems mentioned above.

[0005] The technical solution adopted in this invention is as follows: A method for joint positioning of vehicle-mounted lidar and visual sensors, comprising the following steps: S1: Unify the LiDAR point cloud and visual image to the same clock source and compensate for external parameter changes caused by vehicle vibration online. Extract structurally stable points such as local curvature on the point cloud side and extract superpixel block centers on the image side. After projecting onto a spherical coordinate system, find matching pairs that are adjacent across modal spaces and have consistent semantic descriptions as high-confidence heterogeneous feature seeds.

[0006] S2: Using the heterogeneous feature seeds from S1 as priors, instance-level semantic segmentation is performed on the image, and inconsistent motion clumps are detected in the point cloud. A binary dynamic mask is jointly generated. This mask is used to remove seeds from S1 that fall into dynamic regions, and the results are fed back to update S1.

[0007] S3: The background skeleton nodes of S2 are attached with semantic tags and laser reflection intensity, and fused into a semantic-geometric joint descriptor. Edges are constructed based on co-view relationships and relative poses to form a local semantic topology graph. The topology graphs of the current frame and historical frames are stitched together by a sliding window to maintain a spatially continuous local map.

[0008] S4: Construct a lightweight conditional generation model as a cross-modal illusion field. When lidar or vision fails due to degradation conditions such as nighttime, rain, or fog, the topological node distribution of S3 is used as the spatial control condition, and the static mask of S2 is used to limit the effective area. Based on the point cloud-image joint distribution learned from historical frames, illusionary views or illusionary point clouds are generated in the missing areas.

[0009] S5: Integrate the enhancement seed of S4 and the topological edge constraint of S3 into the factor graph. Add a new type of illusion factor to represent the projection error between illusion features and corresponding map nodes. Its information matrix weights are initialized by the confidence level of S4 and adjusted online according to the actual inlier rate.

[0010] S6: Real-time monitoring of degradation signals such as residual spikes, pose abrupt changes, and illusion factor residual anomalies after S5 optimization. Once localization loss is determined, it no longer relies on the current degradation data, but instead uses the local features continuously perceived by S4 from multiple frames before the loss and the corresponding topological nodes preserved by S3 to construct an illusion memory map. This memory map is then used for global descriptor retrieval with the offline prior map to obtain candidate relocalization poses, which are then back-substituted into the factor map of S5 for verification. When relocalization is successful, this round of illusion memory map is used as a positive sample to reinject into S4, enhancing and updating the generative model.

[0011] In a preferred embodiment, in step S1, the LiDAR and the vision camera are respectively connected to the same IEEE 1588 time server, and the timestamps of the two types of data are aligned to the 1μs level using PPS pulse synchronization. The residual deviation is eliminated by interpolation through a 10-frame circular buffer. For extrinsic parameter calibration, chassis vibration and suspension deformation are continuously monitored during vehicle operation. Using the 200Hz angular velocity and acceleration provided by the IMU, the extrinsic parameter rotation matrix from the LiDAR to the camera is slightly corrected frame by frame. The translation part is estimated online by minimizing the reprojection error of more than 20 static landmarks within a sliding window. On the point cloud side, feature points with curvature exceeding 0.5 and a number of stable neighborhood points greater than 6 are extracted from each scan circle. On the image side, superpixel blocks with high color and gradient consistency are segmented, and their geometric centers are taken as visual feature points. Both types of features are uniformly projected onto a spherical coordinate system with the vehicle center as the origin, the radius is normalized to 1.0, and the azimuth angle is discretized into a grid with a step size of 0.5° and the pitch angle with a step size of 0.3°. Within any grid on the sphere, if it falls simultaneously into both the laser feature projection and the visual feature projection, and the principal direction deviation of their local orientation histograms is less than 8°, it is retained as a pair of heterogeneous feature seeds. These seeds carry a joint label consisting of 3D coordinates, a spherical grid index, a laser reflection intensity gradient, and a 128-dimensional visual local descriptor. This label is then simultaneously passed to S2 as spatial cues for instance segmentation and motion detection, and is treated as an immovable geometric anchor point in the illusion field of S4.

[0012] In a preferred embodiment, in step S2, after receiving the heterogeneous feature seeds transmitted by S1, an instance segmentation network is triggered centered on the landing point of each seed on the image. This network uses a MobileNetV3 backbone paired with a lightweight decoder to output the contour mask of the object containing the seed, with a segmentation confidence threshold of 0.75. Simultaneously, the scene flow between two consecutive frames is calculated on the laser point cloud side, and voxel clumps with a scene flow modulus exceeding 0.3 m / s are marked as regions of inconsistent motion. The image instance mask and the point cloud motion markers are fused according to Bayesian rules to output a binary dynamic mask covering the entire field of view, where pixels belonging to pedestrians, vehicles, and other non-fixed objects are set to 0. This dynamic mask is immediately fed back to S1, and any seeds that fall into the mask value region after 3D coordinate projection are permanently removed.

[0013] In a preferred embodiment, in step S2, after dynamic culling, the background skeleton is extracted from the static scene residual: vertical linear and planar structures are extracted from the laser point cloud, and streetlights, traffic sign poles, guardrail posts, and building edges are locked by the boundary of abrupt changes in reflection intensity. These are then cross-validated with image semantic labels, retaining road signs that are semantically consistent and appear consecutively for more than 5 frames as skeleton nodes. Each skeleton node is assigned 3D coordinates, a semantic category, and a 3×3 covariance matrix, and these are concatenated into skeleton edges according to spatial proximity to form the background topological skeleton. This skeleton serves as the node library input for constructing the semantic topological graph in S3, and the static binary mask generated by the skeleton convex hull projection is passed to S4. When the value of the region inside the mask is 1, illusion generation is possible; when the value of the region outside the mask is 0, complete prohibition is achieved.

[0014] In a preferred embodiment, in step S3, based on the background topology skeleton nodes passed in S2, a 16-dimensional embedding vector is retrieved from the semantic tags for each node. Simultaneously, the laser reflection intensity statistics within a 1.0m radius sphere neighborhood around the node are aggregated, and the embedding vector, intensity mean and variance, and node geometric coordinates are compressed into a 32-dimensional joint descriptor. Edge construction between nodes follows two principles: two nodes directly co-seeing within the current frame's observation range are automatically connected, with edge weights recording relative pose and co-seeing frequency; node pairs with stable co-seeing relationships verified through multiple observations in historical frames are also added to the connection, with edge weights labeled with long-term co-seeing confidence. The generated single-frame topology map is connected to a sliding window, with the window length adaptively adjusted according to travel distance. In low-speed urban scenes, frames within the most recent 200m are retained, while in high-speed scenes, this is extended to 500m.

[0015] In a preferred embodiment, in step S3, for each new frame entering the window, a subgraph isomorphic to the old frame is found through node descriptor matching and graph cut algorithms. After least squares alignment, these subgraphs are stitched together to form a continuous local map. The old node descriptors are updated and fused with new observations using Bayesian methods. The topology graph is synchronously output to both ends: providing node and edge constraints in the factor graph to S5, with each edge introducing a relative pose factor; and providing the spatial distribution and geometric descriptors of active nodes to S4 as input for generating topological conditions in the illusion field. Illusion features can only be generated within a 2.0m radius centered on the active nodes.

[0016] In a preferred embodiment, in step S4, a lightweight conditional energy optimization framework model is pre-trained on historical cross-weather paired data, consisting of two components:

[0017] The first is a fully convolutional generator G, specifically a 5-layer U-Net, whose input is the LiDAR depth projection map D. lidar ∈R H×W×1 The output is the visual feature map F∈R corresponding to the viewpoint. H×W×CWith 128 feature channels (C), this network freezes the joint distribution of LiDAR geometry to visual texture in historical frames.

[0018] The second is the topological feature predictor Φ, which is implemented using a graph convolutional network. The input is the geometric descriptor g of the S3 active semantic topological node set N={ni}. i The pose ξ is used to output a tensor of the same size as F, representing the desired visual features of the topology. When the camera fails at night or in dense fog, the system takes the current frame's LiDAR projection depth map D and at least 5 active nodes within the sliding window in S3 as inputs. Within the effective region constrained by the static binary mask Mstatic provided by S2, the system solves for the final illusion feature map F by iteratively optimizing a total energy function consisting of three weighted sub-energys online. hall ∗ The three sub-energy terms play different roles: the prior maintenance term ensures that the fantasy result does not deviate from the modal mapping learned from history; the topology control term uses node distribution to transform into spatial control signals and constrains the fantasy features to conform to the expected pattern of the topological structure within the skeleton region; and the anchor alignment term uses the S1 heterogeneous seed as an anchor to ensure that the local descriptor of the generated feature at the seed projection position is cosine similar to the seed-invariant feature. The three weighting coefficients are dynamically assigned based on the current degradation level and the seed inliquity rate, and are initially all 1.0.

[0019] The formula for anchor alignment is: ; In the formula: k is the seed index for traversing S1. The seed projection coordinates; To extract F The descriptor after 3x3 region pooling centered on; This is a seed cross-modal invariant feature; For cosine similarity, Huber is a robust kernel function with a threshold set to 0.1. This parameter forces the fantasy feature to align with the real geometric semantic anchor at the seed location.

[0020] In a preferred embodiment, in step S4, online optimization employs gradient descent with momentum, initializing the forward result of the current depth map D from the generator G, with a fixed step size of 0.05 and momentum of 0.9, unfolding 20 steps, and re-evaluating the mask and projection in each step. The topology condition graph generation process required for the topology control term is as follows: For each activated node ni, its 3D coordinates Xi are first mapped to the feature image plane via spherical projection using the current pose ξ in S5. Then, the relative offset between the projected coordinates and the feature map pixel grid is calculated, and after sinusoidal position encoding, it is concatenated with the geometric descriptor gi, fed into a graph convolutional layer with shared weights, and finally weighted and summed using Mahalanobis distance weights and activated by Sigmoid. Another role of pose ξ is reflected in the calculation of the projection coordinates of the heterogeneous seed in the anchor point alignment term S1, using the virtual camera intra-participant current transformation matrix after spatiotemporal alignment to project the seed's 3D coordinates onto the feature image plane. Each time the pose ξ is updated via the S5 sliding window optimization, the projected coordinates are immediately updated, triggering a chain reaction that recalculates the anchor alignment term and the topological condition graph, forcing the illusion field to be re-solved following the latest pose. When heavy rain causes severe attenuation of the LiDAR while the camera remains usable, the roles are reversed: the input becomes the image feature map I, the generator G switches to a symmetric U-Net that learns the image-to-depth mapping, the norm metric in the energy function is adjusted to the depth domain, and the predictor Φ outputs the desired depth distribution. The remaining topology, mask, and seed constraints remain completely consistent. The final generated illusion visual feature map or illusion depth map is injected into the S1 seed set with pixel-level confidence, while simultaneously writing back the S3 node descriptor.

[0021] Formula 1: The diffusion formula for topological condition graphs is: ; In the formula: i represents the number of active nodes to be traversed. For node geometry descriptors, This indicates that the 3D coordinates of the nodes are represented by the pose ξ. The coordinates of the spherical projection onto the feature image plane; For feature map pixel coordinate grid, PE stands for sinusoidal position code; GCN stands for graph convolution operation; It is a spatial weight based on Mahalanobis distance. Activated by Sigmoid. This formula transforms discrete topological nodes into a dense spatial control graph.

[0022] The formula for the pose anchoring projection operator is: ; Parameter definition: This is the 3x4 intrinsic parameter matrix of the virtual camera after spatiotemporal alignment; The Lie algebra of the vehicle pose estimated for S5; Map the Lie algebra to the SE(3) transformation matrix; Let S5 be the 3D homogeneous coordinates of the seed point. This operator directly injects the pose of S5 into the geometric alignment of the fantasy field, achieving optimized loop closure.

[0023] In a preferred embodiment, in step S5, the factor graph contains three types of factors: a standard reprojection factor connecting the real sensor observations with the corresponding map points; a topological edge factor derived from the relative pose constraints between nodes provided in S3; and a fantasy factor connecting the illusion features of S4 with the projection errors of the map nodes in S3, with its initial weights in the information matrix being inversely proportional to the pixel-wise confidence generated during the output of S4. At the start of each iteration, the sliding window optimizer first updates the anchor points of the illusion field in S4 using the pose from the previous frame, triggering the generation of new enhancement seeds and the refresh of node descriptors. Then, it uses the updated factor graph to perform a Gauss-Newton nonlinear least squares solution, with a maximum of 8 iterations.

[0024] In a preferred embodiment, in step S6, at the end of each sliding window optimization, the system simultaneously records the overall residual after optimization, the jump distance of the vehicle pose relative to the previous frame, and the proportion of the illusion factor residual to the total residual. When the overall residual abnormally increases and the pose jump exceeds 1.2m or 2.5°, or the proportion of the illusion factor residual exceeds 50% for 6 consecutive frames, the current localization is determined to have entered a degradation state. Once the degradation determination is triggered, the optimization output of the current frame is immediately frozen, and the local illusion feature maps generated and retained in S4 within the 15 consecutive frames before the degradation occurred are read back and aligned one by one with the semantic topology node snapshots retained in S3 during the same period to stitch together an illusion memory map covering the scanned area before degradation. This memory map contains illusion visual features and joint descriptors of topology nodes. The most similar position is found in the offline prior map through bag-of-words global search, and 3 candidate relocalization poses are returned. Each candidate pose is re-input into the factor map of S5. A validation factor is constructed using the features regenerated in the current frame of S4 and the map points corresponding to the candidate positions. The candidate with the highest inlier rate (over 0.65) is considered successfully relocalized. After successful relocalization, all regions of the phantom memory map involved in the stitching during this degradation event are marked as positive samples, paired with their corresponding real sensor observations, and fed back to the generator in S4 for 500-step incremental fine-tuning at a learning rate of 0.001.

[0025] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0026] 1. In this invention, by introducing a cross-modal illusion field, when extreme conditions cause severe failure of either the LiDAR or the visual camera, the illusion field does not simply discard the damaged data. Instead, it utilizes historical joint distribution priors, the semantic topological skeleton of the current scene, and high-confidence heterogeneous anchor points to generate illusion features in static regions that are highly consistent with the real modality. In energy optimization, the prior maintenance term ensures that the illusion results follow the physical laws of cross-modal mapping; the topology control term restricts the illusion to long-term stable spatial structures such as lampposts and curbs; and the anchor point alignment term forces local features to maintain geometric semantic consistency with the immutable geometric seed. This triple constraint adjusts the spatial anchoring parameters of the illusion field in real time through pose updates, ensuring that the illusion features are always precisely aligned with the current vehicle's viewpoint. Furthermore, the improved enhancement seed significantly increases the convergence radius of the next pose optimization, effectively reducing the risk of positioning jumps and interior point collapses, and greatly improving the autonomous driving capability under complex lighting and weather conditions without intervention.

[0027] 2. In this invention, local memories of multiple consecutive frames of hallucination field output before degradation are traced back and stitched with historical semantic topology nodes to form a hallucination memory graph. Although this graph is composed of imagination, it carries stable descriptors that can be matched with offline prior maps, enabling successful global relocalization. After the pose is recovered, the hallucination samples that successfully guided relocalization are fed back into the generative model for fine-tuning, substantially enhancing the generation quality of the hallucination field in such dangerous modes. This not only provides a soft recovery method when localization fails, but also continuously pushes the performance boundaries of perception and localization systems in long-term deployments through an evolutionary approach oriented towards lifelong learning, providing a low-cost, robust solution for the large-scale deployment of all-weather, all-scenario autonomous driving. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the process principle of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0030] Reference Figure 1 A method for joint localization using vehicle-mounted LiDAR and visual sensors includes the following steps:

[0031] S1: Unify the LiDAR point cloud and visual image to the same clock source and compensate for extrinsic parameter changes caused by vehicle vibration online. Extract structurally stable points such as local curvature on the point cloud side and extract superpixel block centers on the image side. After projecting onto a spherical coordinate system, find matching pairs that are adjacent across modal spaces and have consistent semantic descriptions as high-confidence heterogeneous feature seeds. This seed set provides spatial cues for the dynamic mask in S2 and provides immutable geometric anchors for the illusion generation in S4.

[0032] S2: Using the heterogeneous feature seeds from S1 as priors, instance-level semantic segmentation is performed on the image, and inconsistent motion clumps are detected in the point cloud. A binary dynamic mask is jointly generated. This mask is used to remove seeds from S1 that fall into dynamic regions, and the results are fed back to S1 for updates. For the removed static scene, long-term stable landmarks such as lampposts, curbs, and signpost bases are extracted to form a background topological skeleton, which is output to S3 as node candidates and simultaneously passed to S4 to limit the effective area for illusion generation.

[0033] S3: The background skeleton nodes of S2 are attached with semantic labels and laser reflection intensity, fused into a semantic-geometric joint descriptor, and edges are constructed based on co-view relationships and relative poses to form a local semantic topology graph. The topology graphs of the current frame and historical frames are stitched together by a sliding window to maintain a spatially continuous local map. This topology graph acts as a factor graph constraint in the pose optimization of S5, and its node distribution provides spatial conditions and extent boundaries for the fantasy field of S4.

[0034] S4: Construct a lightweight conditional generation model as a cross-modal illusion field. When LiDAR or vision fails due to degradation conditions such as nighttime, rain, or fog, the topological node distribution of S3 is used as the spatial control condition, and the static mask of S2 is used to define the effective area. Based on the point cloud-image joint distribution learned from historical frames, illusionary views or illusionary point clouds are generated in the missing areas. For example, daytime-like visual features are generated from LiDAR geometry at night, and dense depth is generated from image texture during heavy rain. The generated results must not conflict geometrically or semantically with the heterogeneous seed of S1. Illusionary features are injected into the seed set of S1 to form an enhancement seed and update the node descriptor of S3. The current pose of S5 adjusts the illusion field anchoring parameters, while the enhancement seed, in turn, expands the convergence radius of the next optimization of S5.

[0035] S5: Integrate the enhancement seed from S4 and the topological edge constraints from S3 into the factor graph. Add a new type of illusion factor to represent the projection error between illusion features and corresponding map nodes. Its information matrix weights are initialized by the confidence level from S4 and adjusted online based on the actual inlier rate. Jointly optimize the spatial anchoring parameters of vehicle pose, map node positions, and the S4 illusion field within a sliding window, enabling adaptive fusion of real sensor factors and illusion factors.

[0036] S6: Real-time monitoring of degradation signals such as residual spikes, pose abrupt changes, and illusion factor residual anomalies after S5 optimization. Once localization loss is determined, it no longer relies on the current degradation data, but instead uses the local features continuously perceived by S4 from multiple frames before the loss and the corresponding topological nodes preserved by S3 to construct an illusion memory map. This memory map is then used for global descriptor retrieval with the offline prior map to obtain candidate relocalization poses, which are then back-substituted into the factor map of S5 for verification. When relocalization is successful, this round of illusion memory map is used as a positive sample to reinject into S4, enhancing and updating the generative model.

[0037] In step S1, the LiDAR and visual camera are connected to the same IEEE 1588 time server. The timestamps of the two types of data are aligned to the 1μs level using PPS pulse synchronization. Residual deviations are eliminated through 10-frame circular buffer interpolation. For extrinsic parameter calibration, chassis vibration and suspension deformation are continuously monitored during vehicle operation. Using the 200Hz angular velocity and acceleration provided by the IMU, the extrinsic parameter rotation matrix from the LiDAR to the camera is slightly corrected frame by frame. For translation, the reprojection error of more than 20 static landmarks within a sliding window is minimized online. On the point cloud side, feature points with curvature exceeding 0.5 and a neighborhood of more than 6 stable points are extracted from each scan circle. On the image side, superpixel blocks with high color and gradient consistency are segmented, and their geometric centers are used as visual feature points. Both types of features are projected onto a spherical coordinate system with the vehicle center as the origin. The radius is normalized to 1.0, and the azimuth angle is discretized into a grid with a step size of 0.5° and the pitch angle with a step size of 0.3°. Within any grid on the sphere, if it falls simultaneously into both the laser feature projection and the visual feature projection, and the principal direction deviation of their local orientation histograms is less than 8°, it is retained as a pair of heterogeneous feature seeds. These seeds carry a joint label consisting of 3D coordinates, a spherical grid index, a laser reflection intensity gradient, and a 128-dimensional visual local descriptor. This label is then simultaneously passed to S2 as spatial cues for instance segmentation and motion detection, and is treated as an immovable geometric anchor point in the illusion field of S4.

[0038] In step S2, after receiving the heterogeneous feature seeds transmitted from S1, an instance segmentation network is triggered centered on the landing point of each seed on the image. This network uses a MobileNetV3 backbone paired with a lightweight decoder to output the contour mask of the object containing the seed, with a segmentation confidence threshold of 0.75. Simultaneously, the scene flow between two consecutive frames is calculated on the laser point cloud side, and voxel clumps with a scene flow modulus exceeding 0.3 m / s are marked as regions of inconsistent motion. The image instance mask and the point cloud motion markers are fused according to Bayesian rules to output a binary dynamic mask covering the entire field of view, where pixels belonging to pedestrians, vehicles, and other non-fixed objects are set to 0. This dynamic mask is immediately fed back to S1, and any seeds that fall into the mask value of 0 after 3D coordinate projection are permanently removed.

[0039] In step S2, after dynamic culling, the background skeleton is extracted from the static scene residuals: vertical linear and planar structures are extracted from the laser point cloud, and streetlights, traffic sign poles, guardrail posts, and building edges are identified by the boundary of abrupt changes in reflection intensity. These are then cross-validated with the image semantic labels, retaining road signs that are semantically consistent and appear consecutively for more than 5 frames as skeleton nodes. Each skeleton node is assigned 3D coordinates, a semantic category, and a 3×3 covariance matrix, and these are concatenated into skeleton edges based on spatial proximity to form the background topological skeleton. This skeleton serves as the node library input for constructing the semantic topological graph in S3, and the static binary mask generated by the skeleton's convex hull projection is passed to S4. When the value inside the mask is 1, illusion generation is allowed; when the value outside the mask is 0, it is completely disabled.

[0040] In step S3, based on the background topology skeleton nodes passed in from S2, a 16-dimensional embedding vector is retrieved from the semantic tags for each node. Simultaneously, the laser reflection intensity statistics within a 1.0m radius sphere neighborhood around the node are aggregated. The embedding vector, intensity mean and variance, and node geometric coordinates are compressed into a 32-dimensional joint descriptor. Edge construction between nodes follows two principles: two nodes directly co-seeing within the current frame's observation range are automatically connected, with edge weights recording relative pose and co-seeing frequency; node pairs with stable co-seeing relationships verified through multiple observations in historical frames are also added to the connection, with edge weights labeled with long-term co-seeing confidence. The generated single-frame topology map is connected to a sliding window, with the window length adaptively adjusted according to travel distance. In low-speed urban scenes, frames within the most recent 200m are retained, while in high-speed scenes, this is extended to 500m.

[0041] In step S3, for each new frame entering the window, a subgraph isomorphic to the old frame is found through node descriptor matching and graph cut algorithms. After least squares alignment, these subgraphs are stitched together to form a continuous local map. The old node descriptors are updated and fused with new observations using Bayesian methods. The topology graph is output synchronously to both ends: providing node and edge constraints from the factor graph to S5, with each edge introducing a relative pose factor; and providing the spatial distribution and geometric descriptors of active nodes to S4, serving as input for generating topological conditions in the illusion field. Illusion features can only be generated within a 2.0m radius centered on the active nodes.

[0042] In step S4, a lightweight conditional energy optimization framework model is pre-trained on historical cross-weather paired data, with two components:

[0043] The first is a fully convolutional generator G, specifically a 5-layer U-Net, whose input is the LiDAR depth projection map D. lidar ∈R H×W×1 The output is the visual feature map F∈R corresponding to the viewpoint. H×W×C With 128 feature channels (C), this network freezes the joint distribution of LiDAR geometry to visual texture in historical frames.

[0044] The second is the topological feature predictor Φ, which is implemented using a graph convolutional network. The input is the geometric descriptor g of the S3 active semantic topological node set N={ni}. i The pose ξ is used to output a tensor of the same size as F, representing the desired visual features of the topology. When the camera fails at night or in dense fog, the system takes the current frame's LiDAR projection depth map D and at least 5 active nodes within the sliding window in S3 as inputs. Within the effective region constrained by the static binary mask Mstatic provided by S2, the system solves for the final illusion feature map F by iteratively optimizing a total energy function consisting of three weighted sub-energys online. hall ∗ The three sub-energy terms play different roles: the prior maintenance term ensures that the fantasy result does not deviate from the modal mapping learned from history; the topology control term uses node distribution to transform into spatial control signals and constrains the fantasy features to conform to the expected pattern of the topological structure within the skeleton region; and the anchor alignment term uses the S1 heterogeneous seed as an anchor to ensure that the local descriptor of the generated feature at the seed projection position is cosine similar to the seed-invariant feature. The three weighting coefficients are dynamically assigned based on the current degradation level and the seed inliquity rate, and are initially all 1.0.

[0045] The formula for anchor alignment is: ; In the formula: k is the seed index for traversing S1. The seed projection coordinates; To extract F The descriptor after 3x3 region pooling centered on; This is a seed cross-modal invariant feature; For cosine similarity, Huber is a robust kernel function with a threshold set to 0.1. This parameter forces the fantasy feature to align with the real geometric semantic anchor at the seed location.

[0046] In step S4, online optimization employs gradient descent with momentum, initializing the forward result of the current depth map D from the generator G. The step size is fixed at 0.05, the momentum at 0.9, and the process unfolds for 20 steps, re-evaluating the mask and projection at each step. The topology condition graph generation process required for the topology control term is as follows: For each activated node ni, its 3D coordinates Xi are first mapped to the feature image plane via spherical projection using the current pose ξ in S5. Then, the relative offset between the projected coordinates and the feature map pixel grid is calculated, sinusoidally encoded, concatenated with the geometric descriptor gi, and fed into a graph convolutional layer with shared weights. Finally, a weighted sum is obtained using Mahalanobis distance weights and activated via Sigmoid. Another role of pose ξ is reflected in the calculation of the projected coordinates of the heterogeneous seed in the anchor alignment term S1. Using the virtual camera's intra-transformation matrix after spatiotemporal alignment, the seed's 3D coordinates are projected onto the feature image plane. Each time the pose ξ is updated in the S5 sliding window optimization, the projected coordinates are immediately updated, triggering a chain reaction that recalculates the anchor alignment term and the topology condition graph, forcing the illusion field to be re-solved following the latest pose. When heavy rain causes severe degradation of the LiDAR while the camera remains usable, the roles are reversed: the input becomes the image feature map I, the generator G switches to a symmetric U-Net that learns the image-to-depth mapping, the norm metric in the energy function is adjusted to the depth domain, and the predictor Φ outputs the desired depth distribution. The remaining topology, mask, and seed constraints remain completely consistent. The final generated illusion visual feature map or illusion depth map is injected into the S1 seed set with pixel-level confidence, while simultaneously writing back the S3 node descriptor.

[0047] The diffusion formula for the topological condition graph is: ; In the formula: i represents the number of active nodes to be traversed. For node geometry descriptors, This indicates that the 3D coordinates of the nodes are represented by the pose ξ. The coordinates of the spherical projection onto the feature image plane; For feature map pixel coordinate grid, PE stands for sinusoidal position code; GCN stands for graph convolution operation; It is a spatial weight based on Mahalanobis distance. Activated by Sigmoid. This formula transforms discrete topological nodes into a dense spatial control graph.

[0048] The formula for the pose anchoring projection operator is: ; Parameter definition: This is the 3x4 intrinsic parameter matrix of the virtual camera after spatiotemporal alignment; The Lie algebra of the vehicle pose estimated for S5; Map the Lie algebra to the SE(3) transformation matrix; Let S5 be the 3D homogeneous coordinates of the seed point. This operator directly injects the pose of S5 into the geometric alignment of the fantasy field, achieving optimized loop closure.

[0049] In step S5, the factor graph contains three types of factors: standard reprojection factors connect real sensor observations with corresponding map points; topological edge factors come from the relative pose constraints between nodes provided in S3; and illusion factors connect the illusion features of S4 with the projection errors of map nodes in S3, with the initial weight of its information matrix being the inverse of the pixel-wise confidence generated during the output of S4. At the beginning of each iteration, the sliding window optimizer first uses the pose of the previous frame to update the anchor points of the illusion field in S4, triggering the generation of new enhancement seeds and the refresh of node descriptors. Then, it uses the updated factor graph to perform Gauss-Newton nonlinear least squares solution, with a maximum of 8 iterations.

[0050] In step S6, at the end of each sliding window optimization, the system simultaneously records the overall residual after optimization, the jump distance of the vehicle pose relative to the previous frame, and the proportion of the illusion factor residual to the total residual. When the overall residual abnormally increases and the pose jump exceeds 1.2m or 2.5°, or the proportion of the illusion factor residual exceeds 50% for 6 consecutive frames, the current localization is determined to have entered a degraded state. Once the degradation determination is triggered, the optimization output of the current frame is immediately frozen, and the local illusion feature maps generated and retained in S4 within the 15 consecutive frames before the degradation occurs are read back and aligned one by one with the semantic topology node snapshots retained in S3 during the same period to stitch together an illusion memory map covering the scanned area before degradation. This memory map contains illusion visual features and joint descriptors of topology nodes. The most similar position is found in the offline prior map through bag-of-words global search, and 3 candidate relocalization poses are returned. Each candidate pose is re-input into the factor map of S5, and a verification factor is constructed using the features regenerated in S4 of the current frame and the map points corresponding to the candidate positions. The candidate with the highest inlier rate and an inlier rate exceeding 0.65 is considered to have successfully relocalized. After successful relocation, all the fantasy memory map regions involved in the splicing during this degradation event were marked as positive samples, paired with the corresponding real sensor observations, and fed back to the generator of S4 for 500-step incremental fine-tuning at a learning rate of 0.001.

[0051] In this invention, by introducing a cross-modal illusion field, when extreme conditions cause severe failure of either the LiDAR or the visual camera, the illusion field does not simply discard the damaged data. Instead, it utilizes historical joint distribution priors, the semantic topological skeleton of the current scene, and high-confidence heterogeneous anchor points to generate illusion features in static regions that are highly consistent with the real modality. In energy optimization, the prior maintenance term ensures that the illusion results follow the physical laws of cross-modal mapping; the topology control term confines the illusion to long-term stable spatial structures such as lampposts and curbs; and the anchor point alignment term forces local features to maintain geometric semantic consistency with the immutable geometric seed. This triple constraint adjusts the spatial anchoring parameters of the illusion field in real time through pose updates, ensuring that the illusion features are always precisely aligned with the current vehicle's viewpoint. Furthermore, the improved enhancement seed significantly increases the convergence radius of the next pose optimization, effectively reducing the risk of positioning jumps and interior point collapse, and greatly improving the autonomous driving capability for uninterrupted operation under complex lighting and weather conditions.

[0052] In this invention, local memories of the hallucination field output across multiple consecutive frames before degradation are traced back and stitched together with historical semantic topology nodes to form a hallucination memory graph. Although this graph is composed of imagination, it carries stable descriptors that match the offline prior map, enabling successful global relocalization. After the pose is recovered, the successfully relocalized hallucination samples are fed back into the generative model for fine-tuning, substantially enhancing the generation quality of the hallucination field in such dangerous modes. This not only provides a soft recovery mechanism when localization fails but also continuously pushes the performance boundaries of perception and localization systems in long-term deployments through an evolutionary approach oriented towards lifelong learning, providing a low-cost, robust solution for the large-scale deployment of all-weather, all-scenario autonomous driving.

[0053] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0054] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for joint positioning using vehicle-mounted lidar and visual sensors, characterized in that, Includes the following steps: S1: Unify the LiDAR point cloud and visual image to the same clock source and compensate for the external parameter changes caused by vehicle vibration online; extract structurally stable points such as local curvature on the point cloud side, extract the superpixel block center on the image side, project them onto the spherical coordinate system, and find matching pairs that are adjacent across modal spaces and have consistent semantic descriptions as high-confidence heterogeneous feature seeds. S2: Using the heterogeneous feature seeds of S1 as priors, perform instance-level semantic segmentation on the image, detect inconsistent motion clumps in the point cloud, and jointly generate a binary dynamic mask; use this mask to remove the seeds in S1 that fall into the dynamic region, and feed them back to update S1. S3: Add semantic labels and laser reflection intensity to the background skeleton nodes of S2 and fuse them into a semantic-geometric joint descriptor. Construct edges based on co-view relationships and relative poses to form a local semantic topology graph. Stitch the topology graphs of the current frame and historical frames through a sliding window to maintain a spatially continuous local map. S4: Build a lightweight conditional generation model as a cross-modal illusion field; when lidar or vision fails due to degradation conditions such as night, rain and fog, use the topological node distribution of S3 as spatial control conditions, use the static mask of S2 to limit the effective area, and generate illusion views or illusion point clouds in the missing area based on the point cloud-image joint distribution learned from historical frames. S5: Unify the enhancement seed of S4 and the topological edge constraint of S3 into the factor graph; add a new type of illusion factor to represent the projection error between illusion features and corresponding map nodes. Its information matrix weight is initialized by the confidence of S4 and adjusted online according to the actual inlier rate. S6: Real-time monitoring of degradation signals such as residual abrupt increase, pose change, and fantasy factor residual abnormality after S5 optimization; Once localization is determined to be lost, the system no longer relies on the current degraded data. Instead, it uses the local features obtained from multiple consecutive illusions in S4 before the loss and the corresponding topological nodes preserved in S3 to create an illusion memory map. This memory map is then used to perform a global descriptor search with the offline prior map to obtain candidate relocalization poses. These poses are then back-substituted into the factor map of S5 for verification. If relocalization is successful, this round of illusion memory map is used as a positive sample to feed back into S4 and enhance and update the generative model.

2. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S1, the lidar and vision camera are respectively connected to the same IEEE1588 time server. The timestamps of the two types of data are aligned to the 1μs level using PPS pulse synchronization. The residual deviation is eliminated by interpolation through a 10-frame circular buffer. For external parameter calibration, chassis vibration and suspension deformation are continuously monitored during vehicle operation. The external parameter rotation matrix from lidar to camera is slightly corrected frame by frame using the 200Hz angular velocity and acceleration provided by the IMU. The translation part is estimated online by minimizing the reprojection error of more than 20 static landmarks within a sliding window. On the point cloud side, feature points with curvature greater than 0.5 and more than 6 stable neighborhood points are extracted from each scan circle. On the image side, superpixel blocks with high color and gradient consistency are segmented, and their geometric centers are taken as visual feature points. Both types of features are uniformly projected onto a spherical coordinate system with the vehicle center as the origin. The radius is normalized to 1.0, and the azimuth angle is discretized into a grid with a step size of 0.5° and the pitch angle with a step size of 0.3°. If any grid on the sphere falls into both the laser feature projection and the visual feature projection, and the main direction deviation of their local direction histograms is less than 8°, it is retained as a pair of heterogeneous feature seeds. These seeds carry a joint label consisting of three-dimensional coordinates, a spherical grid index, a laser reflection intensity gradient, and a 128-dimensional visual local descriptor. They are then simultaneously passed to S2 as spatial cues for instance segmentation and motion detection, and are regarded as immovable geometric anchors in the illusion field of S4.

3. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S2, after receiving the heterogeneous feature seeds transmitted by S1, an instance segmentation network is triggered with the landing point of each seed on the image as the center. This network uses a MobileNetV3 backbone with a lightweight decoder to output the contour mask of the object where the seed is located, and the segmentation confidence threshold is set to 0.

75. At the same time, the scene flow between two consecutive frames is calculated on the laser point cloud side, and voxel clumps with a scene flow modulus exceeding 0.3m / s are marked as regions with inconsistent motion. The image instance mask and the point cloud motion markers are fused according to Bayesian rules to output a binary dynamic mask covering the entire field of view. Pixels belonging to pedestrians, vehicles and other non-fixed objects are set to 0. This dynamic mask is immediately fed back to S1, and any seeds that fall into the mask value of 0 after 3D coordinate projection are permanently removed.

4. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S2, after dynamic culling is completed, the background skeleton is extracted from the static scene residual: vertical linear and planar structures are extracted from the laser point cloud, and street lamp poles, traffic sign poles, guardrail posts and building edges are locked by the boundary of abrupt change in reflection intensity. Then, cross-validation is performed with the image semantic labels, and road signs that are semantically consistent and appear continuously for more than 5 frames are retained as skeleton nodes. Each skeleton node is attached with three-dimensional coordinates, semantic category and 3×3 covariance matrix, and strung together according to spatial proximity to form skeleton edges, forming the background topological skeleton. This skeleton is used as the node library input for constructing the semantic topological graph in S3, and the static binary mask generated by the convex hull projection of the skeleton is passed to S4. When the value of the region inside the mask is 1, illusion generation can be performed, and when the value of the region outside the mask is 0, it is completely prohibited.

5. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S3, based on the background topology skeleton nodes passed in from S2, a 16-dimensional embedding vector is retrieved from the semantic tags for each node. At the same time, the laser reflection intensity statistics within a 1.0m radius sphere neighborhood around the node are aggregated, and the embedding vector, intensity mean and variance, and node geometric coordinates are compressed into a 32-dimensional joint descriptor. The edge construction between nodes follows two principles: two nodes that are directly co-seeing within the observation range of the current frame are automatically connected, and the edge weight records the relative pose and the number of co-seeing times; node pairs that have been verified to have a stable co-seeing relationship through multiple observations in historical frames are also added to the connection, and the edge weight is labeled with the long-term co-seeing confidence. The generated single-frame topology map is connected to a sliding window, and the window length is adaptive according to the driving distance. In urban low-speed scenes, frames within the most recent 200m are retained, and in high-speed scenes, the window length is extended to 500m.

6. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S3, for each new frame that enters the window, a subgraph isomorphic to the old frame is found through node descriptor matching and graph cut algorithm. After least squares alignment, the subgraphs are stitched together to form a continuous local map. The old node descriptors are updated and fused with new observations according to Bayesian algorithm. The topology graph is output synchronously to both ends: it provides node and edge constraints in the factor graph to S5, and introduces a relative pose factor for each edge; it provides the spatial distribution and geometric descriptor of active nodes to S4 as input for the generation of topological conditions in the fantasy field. Fantasy features can only be generated within a 2.0m radius centered on the active nodes.

7. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S4, a lightweight conditional energy optimization framework model was pre-trained on historical cross-weather paired data, with two components: The first is a fully convolutional generator G, specifically a 5-layer U-Net, whose input is the LiDAR depth projection map D. lidar ∈R H ×W×1 The output is the visual feature map F∈R corresponding to the viewpoint. H×W×C With 128 feature channels C, this network freezes the joint distribution of LiDAR geometry to visual texture in historical frames; The second is the topological feature predictor Φ, which is implemented using a graph convolutional network. The input is the geometric descriptor g of the S3 active semantic topological node set N={ni}. i The system outputs a tensor of the same size as F, along with the pose ξ, representing the desired visual features of the topology. When the camera fails at night or in dense fog, the system takes the current frame's LiDAR projection depth map D and at least 5 active nodes within the sliding window in S3 as inputs. Within the effective region constrained by the static binary mask Mstatic provided by S2, it iteratively optimizes an online total energy function consisting of three weighted sub-energys to solve for the final illusion feature map F. hall ∗ The three sub-energy terms play different roles: the prior maintenance term forces the fantasy result to not deviate from the modal mapping learned from history; the topology control term uses the node distribution to transform into a spatial control signal and constrains the fantasy features to obey the expected pattern of the topology within the skeleton region; and the anchor alignment term uses the S1 heterogeneous seed as an anchor to ensure that the local descriptor of the generated feature at the seed projection position is cosine similar to the seed invariant feature. The three weighting coefficients are dynamically assigned based on the current degradation level and the seed in-point rate, and are all initially set to 1.

0. The formula for anchor alignment is: In the formula: k is the seed index for traversing S1. Seed projection coordinates; To extract F The descriptor after 3x3 region pooling centered on; This is a seed cross-modal invariant feature; For cosine similarity, Huber is a robust kernel function with a threshold set to 0.1; this item forces the fantasy feature to align with the real geometric semantic anchor at the seed position.

8. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S4, online optimization uses gradient descent with momentum to initialize the forward result of the current depth map D from the generator G. The step size is fixed at 0.05, the momentum is 0.9, and it unfolds for 20 steps. The mask and projection are re-evaluated in each step. The topology condition graph generation process required by the topology control term is as follows: For each active node ni, its 3D coordinates Xi are first mapped to the feature image plane through spherical projection using the current pose ξ in S5. Then, the relative offset between the projected coordinates and the feature map pixel grid is calculated. After sinusoidal position encoding, it is concatenated with the geometric descriptor gi and fed into the graph convolutional layer with shared weights. Finally, it is weighted and summed with Mahalanobis distance weights and activated by Sigmoid. Another role of pose ξ is reflected in the calculation of the projection coordinates of the heterogeneous seed in the anchor point alignment term S1. The seed's 3D coordinates are projected onto the feature image plane using the virtual camera in the spatiotemporally aligned virtual camera in the current transformation matrix. Each time the pose ξ is updated after the S5 sliding window optimization, the projected coordinates are updated immediately, triggering the recalculation of the anchor point alignment term and the topology condition graph, forcing the fantasy field to be re-solved following the latest pose; When heavy rain causes severe attenuation of the LiDAR while the camera remains usable, the roles are reversed. The input becomes the image feature map I, the generator G is switched to a symmetric U-Net that learns the image-to-depth mapping, the norm metric in the energy function is adjusted to the depth domain, and the predictor Φ outputs the desired depth distribution. The remaining topology, mask, and seed constraints are completely consistent. The final generated illusion visual feature map or illusion depth map is injected into the S1 seed set with pixel-level confidence, and the S3 node descriptor is written back. Formula 1: The diffusion formula for topological condition graphs is: In the formula: i represents the number of active nodes to be traversed. For node geometry descriptors, This indicates that the 3D coordinates of the nodes are represented by the pose ξ. The coordinates of the spherical projection onto the feature image plane; For feature map pixel coordinate grid, PE stands for sinusoidal position code; GCN stands for graph convolution operation; It is a spatial weight based on Mahalanobis distance. Activated with Sigmoid; this formula transforms discrete topological nodes into a dense spatial control graph; The formula for the pose anchoring projection operator is: Parameter definition: This is the 3x4 intrinsic parameter matrix of the virtual camera after spatiotemporal alignment; The Lie algebra of the vehicle pose estimated for S5; Map the Lie algebra to the SE(3) transformation matrix; The seed point has three-dimensional homogeneous coordinates; this operator directly injects the pose of S5 into the geometric alignment of the fantasy field, achieving optimized closed loop.

9. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S5, the factor graph contains three types of factors: standard reprojection factors connect real sensor observations with corresponding map points; topological edge factors come from the relative pose constraints between nodes provided in S3; illusion factors connect the illusion features of S4 with the projection errors of map nodes in S3, and the initial weight of its information matrix is ​​equal to the inverse of the pixel-wise confidence generated when S4 is output; at the beginning of each iteration of the sliding window optimizer, the anchor point of the illusion field of S4 is updated once using the pose of the previous frame, triggering the generation of new enhancement seeds and the refresh of node descriptors, and then the updated factor graph is used to perform Gauss-Newton nonlinear least squares solution, with a maximum number of iterations of 8.

10. The method for joint positioning of vehicle-mounted lidar and vision sensor as described in claim 1, characterized in that: In step S6, at the end of each sliding window optimization, the system simultaneously records the overall residual after optimization, the jump distance of the vehicle pose relative to the previous frame, and the proportion of the illusion factor residual to the total residual. When the overall residual abnormally increases and the pose jump exceeds 1.2m or 2.5°, or the proportion of the illusion factor residual exceeds 50% for 6 consecutive frames, it is determined that the current positioning has entered a degradation state. Once the degradation determination is triggered, the optimization output of the current frame is immediately frozen, and the local illusion feature map generated and retained in S4 within 15 consecutive frames before the degradation occurs is read back and aligned with the semantic topology node snapshot retained in S3 at the same time to stitch together an illusion memory map covering the scanned area before degradation.

Images