A multi-sensor fusion semantic enhancement incremental automatic driving mapping method and system

Abstract

The application discloses a multi-sensor fusion semantic enhancement incremental automatic driving mapping method and system, comprising: acquiring multi-sensor synchronous data and performing multi-modal semantic fusion, generating semantic point cloud and extracting road signs; performing semantic weighted front-end odometry, weighting registration residual according to semantic label attributes, obtaining a pose and a key frame; constructing a unified factor graph containing semantic road sign observation, geometric prior and common view constraint factor for back-end optimization; adopting an incremental smoothing solver for optimization, outputting a global trajectory and a map element; comparing a new road sign with a historical road sign, performing semantic level change detection; based on the detection result, locally re-optimizing an affected area, and fusing an incremental patch to an existing map, and outputting a multi-layer high-precision map. The application embeds semantics into a SLAM full link, realizes centimeter-level precision and supports quasi-real-time updating.

Description

Technical Field

[0001] This invention relates to the field of high-precision map construction technology for autonomous driving, and in particular to a multi-sensor fusion semantic enhancement incremental autonomous driving mapping method and system. Background Technology

[0002] High-precision maps are the core foundation for autonomous driving systems to achieve accurate positioning, path planning, and motion decision-making. Their accuracy and freshness directly determine the safety and reliability of autonomous driving. Currently, the mainstream high-precision map construction method is still dominated by "offline data collection + global optimization + manual annotation": data collection vehicles equipped with LiDAR, RTK-GNSS, and IMU scan back and forth along a predetermined route multiple times, and the collected point cloud data is optimized offline in batch processing. Then, road elements are annotated and vectorized manually or semi-automatically, and finally, a high-precision map is generated and distributed to mass-produced vehicles.

[0003] However, the aforementioned traditional solutions have revealed increasingly acute contradictions in large-scale deployment—the map update speed lags far behind the rate of change in the road environment. Dynamic changes in real-world scenarios, such as road construction, traffic marking repainting, temporary detours, and the installation of new traffic signs, require high-precision maps to have weekly or even daily update capabilities. Traditional solutions, however, have a complete cycle of data collection, transmission, post-processing, and manual annotation that takes weeks to months, making it difficult to meet the basic requirement of map freshness for autonomous driving systems. More importantly, even with increased numbers of data collection vehicles and annotation personnel, the traditional "heavy collection, heavy processing" model, while experiencing a sharp increase in marginal costs, cannot solve the challenges of data association and consistency maintenance in crowdsourced incremental updates—how to efficiently integrate data collected by different vehicles at different times, and how to accurately locate local changes and quickly reflect them in existing maps—these problems lack effective solutions within the traditional global batch processing framework.

[0004] The root of the above problems lies in the fact that traditional mapping methods treat "map creation" as a one-time offline task, rather than a dynamic process that requires continuous maintenance. This technical approach dictates that each map update must re-optimize the entire dataset, making it impossible to reuse existing map results. This leads to a significant waste of computing power on repeatedly processing unchanged areas. Furthermore, as the map size increases, the computational load of global optimization grows exponentially, gradually becoming impractical.

[0005] Meanwhile, existing technologies also have significant shortcomings in pose estimation accuracy. In typical scenarios with degraded geometry—such as long straight structures in tunnels, underground parking lots lacking obvious features, highways with poor texture, and highly repetitive urban elevated roads—SLAM systems driven purely by geometric features (points, lines, and surfaces) experience pose drift due to a lack of sufficient geometric constraints, resulting in a sharp decline in mapping accuracy or even complete failure. Although some research in recent years has attempted to introduce semantic information to assist perception or loop closure detection, semantic labels mostly remain at the level of visual annotation or dynamic object filtering, failing to be deeply embedded in the backend optimization framework. Consequently, this stable, high-dimensional environmental observation signal of semantic information fails to play its due constraining role in pose estimation. Summary of the Invention

[0006] In view of the above-mentioned shortcomings in the current field of high-precision map construction technology for autonomous driving, the present invention provides a multi-sensor fusion semantic enhancement incremental autonomous driving mapping method and system, which can construct a semantic landmark-level incremental update mechanism by deeply embedding semantic information into the entire link of front-end weighting, back-end factor graph optimization and change detection, and achieve centimeter-level mapping accuracy.

[0007] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0008] A multi-sensor fusion semantic enhancement incremental autonomous driving mapping method includes:

[0009] Acquire multi-sensor synchronous data, perform multimodal semantic perception and cross-modal fusion on the synchronous data, generate a unified semantic point cloud with semantic labels, and extract semantic landmarks based on the unified semantic point cloud;

[0010] Based on the unified semantic point cloud and the semantic landmarks, a semantically weighted front-end odometer is executed. Different weights are assigned to the point cloud registration residuals according to the static / dynamic attributes of the semantic tags to obtain the vehicle's initial pose estimation and keyframes.

[0011] A unified factor graph is constructed for backend optimization. The unified factor graph includes at least semantic landmark observation factors, semantic geometric prior factors, and semantic co-visibility constraint factors.

[0012] An incremental smoothing solver is used to optimize the unified factor graph, and the optimized global pose trajectory and semantic map features are output.

[0013] The newly acquired semantic landmarks are compared and correlated with historical semantic landmarks in the existing map to perform semantic landmark-level change detection and identify the change status of the landmarks.

[0014] Based on the change detection results, the affected areas are locally re-optimized, and the optimized map elements are integrated into the existing map in the form of incremental patches to output an updated multi-layer high-precision map.

[0015] According to one aspect of the present invention, the multimodal semantic perception and cross-modal fusion specifically includes:

[0016] A 2D multi-task neural network is used to perform semantic segmentation, panoramic segmentation, object detection, and lane line detection on the surround view image to obtain 2D semantic features; a 3D point cloud semantic segmentation network is used to perform point-by-point semantic segmentation on the LiDAR point cloud to obtain 3D semantic features; and a cross-modal attention mechanism is used to fuse the 2D semantic features and the 3D semantic features point-by-point to generate the unified semantic point cloud.

[0017] The extraction of semantic landmarks includes: clustering instances of rod-shaped objects and signs in the unified semantic point cloud, and reconstructing their three-dimensional spatial position and geometric parameters using cylindrical fitting or planar fitting methods.

[0018] According to one aspect of the present invention, in the semantically weighted front-end odometry, different weights are assigned to the point cloud registration residuals based on the static / dynamic attributes of the semantic tags, specifically including:

[0019] Semantic labels are classified into static high-weight categories, dynamic zero-weight categories, and intermediate-weight categories. Point cloud registration residuals belonging to static land cover categories are assigned a first weight, and point cloud registration residuals belonging to dynamic object categories are assigned a second weight. The first weight is greater than the second weight. The second weight includes zero weight and weights close to zero.

[0020] According to one aspect of the present invention, the unified factor graph includes at least semantic landmark observation factors, semantic geometric prior factors, and semantic co-visibility constraint factors, wherein:

[0021] The semantic landmark observation factor is constructed based on the reprojection residual between the vehicle pose and the spatial location of the semantic landmark. The information matrix of the residual is adaptively weighted according to the semantic category and semantic confidence. The semantic category includes static land feature category and dynamic object category, and different semantic categories correspond to different weight values.

[0022] The semantic geometry prior factors include at least one of the following: the verticality constraint factor of the pole-shaped semantic landmark, the orientation constraint factor of the sign semantic landmark, and the parallel constraint factor between landmarks of the same semantic category.

[0023] The semantic co-view constraint factor is derived from the co-view observation of the same semantic landmark by multiple consecutive keyframes and is used to constrain the relative pose between the multiple keyframes.

[0024] According to one aspect of the present invention, it further includes constructing a five-layer semantic scene graph, wherein the five-layer semantic scene graph, from bottom to top, is: a sensor observation layer, a keyframe pose layer, a semantic landmark and structure layer, a road topology layer, and a global region layer;

[0025] The sensor observation layer stores raw sensor data and extracted semantic features; the keyframe pose layer stores optimized keyframe poses and their covariance; the semantic landmark and structure layer stores the category, location, geometric parameters, and co-view relationships of the semantic landmarks; the road topology layer stores the vectorized topological relationships of lane lines, road boundaries, and intersections; and the global region layer stores region-level scene semantic summary information.

[0026] The construction of the unified factor graph is based on the relationship between each level in the five-layer semantic scene graph, establishing constraint edges between factor nodes and variable nodes.

[0027] According to one aspect of the present invention, the multi-layer high-precision map comprises the following five layers:

[0028] Point cloud layer, storing dense 3D point clouds;

[0029] The semantic landmark layer stores the category, spatial location, geometric parameters, and unique identifier of the semantic landmarks;

[0030] The vectorized road layer stores the vector geometric information of lane lines, road boundaries, and traffic signs;

[0031] Semantic raster layer: The ground is rasterized and semantically labeled according to semantic categories;

[0032] The scene layer stores high-level semantic information from the five-layer semantic scene graph.

[0033] According to one aspect of the present invention, the semantic landmark-level change detection specifically includes:

[0034] The newly acquired semantic landmarks are matched with historical semantic landmarks in the existing map in terms of spatial location and semantic category.

[0035] Based on the matching results, each semantic landmark is determined to be one of the following four change states:

[0036] A new status has been added: newly collected road signs have no matching items in the existing map;

[0037] Removed status; historical road signs in the existing map have no matching entries in the newly collected data.

[0038] The status has been modified; the newly collected landmark successfully matches the historical landmark, but its spatial location or geometric parameters have changed.

[0039] In a stable state, newly collected road signs are successfully matched with historical road signs, and the differences in spatial location and geometric parameters are less than the preset threshold.

[0040] According to one aspect of the present invention, the local re-optimization of the affected area based on the change detection results specifically includes:

[0041] The affected area is determined based on the spatial location of the semantic landmarks that are identified as new, removed, or modified.

[0042] The factor graph is reconstructed and locally optimized only for the pose nodes and semantic landmark nodes within the affected region, while the pose nodes and semantic landmark nodes outside the affected region remain unchanged.

[0043] The locally optimized map features are stitched together with map features from unaffected areas to generate an incremental patch.

[0044] According to one aspect of the invention, prior to performing semantically weighted front-end odometer calculation, a dual-mode gating decision step based on scene complexity is further included:

[0045] Calculate the scene complexity metrics for the current scene, wherein the scene complexity metrics include at least one of the following: point cloud feature entropy, the proportion of dynamic objects in the total number of point clouds, RTK-GNSS signal confidence, and image texture richness.

[0046] When the complexity of the scenario is below the first threshold, switch to fast thinking mode and only perform pose tracking using VIO / LIO odometry based on geometric features;

[0047] When the complexity of the scenario exceeds the second threshold, the system switches to slow thinking mode and triggers semantic reasoning enhancement. The semantic reasoning enhancement includes calling the reasoning chain to generate a structured semantic reasoning result for the current scenario, and adjusting the weight allocation strategy of the semantically weighted front-end odometer and the information matrix configuration of each semantic factor in the unified factor graph based on the reasoning result.

[0048] The first threshold is lower than the second threshold, and the two form a hysteresis interval to avoid frequent mode switching.

[0049] A multi-sensor fusion semantic enhancement incremental autonomous driving mapping system includes:

[0050] The acquisition module acquires multi-sensor synchronous data, performs multimodal semantic perception and cross-modal fusion on the synchronous data, generates a unified semantic point cloud with semantic labels, and extracts semantic landmarks based on the unified semantic point cloud.

[0051] The weighting module, based on the unified semantic point cloud and the semantic landmarks, performs semantic weighted front-end odometer measurement, assigns different weights to the point cloud registration residuals according to the static / dynamic attributes of the semantic tags, and obtains the vehicle's initial pose estimation and keyframes.

[0052] The optimization module constructs a unified factor graph for backend optimization. The unified factor graph includes at least semantic landmark observation factors, semantic geometric prior factors, and semantic co-view constraint factors. The semantic landmark observation factors are used to constrain the reprojection error between the vehicle pose and the semantic landmark. The semantic geometric prior factors are used to constrain the geometric properties of the semantic landmark. The semantic co-view constraint factors are used to constrain the relative pose relationship between multiple keyframes that co-view the same semantic landmark.

[0053] The map module uses an incremental smoothing solver to optimize the unified factor map and outputs the optimized global pose trajectory and semantic map features.

[0054] The comparison module compares the newly acquired semantic landmarks with the historical semantic landmarks in the existing map, performs semantic landmark-level change detection, and identifies the change status of the landmarks.

[0055] The patch module, based on the change detection results, performs local re-optimization on the affected areas, integrates the optimized map elements into the existing map in the form of incremental patches, and outputs an updated multi-layer high-precision map.

[0056] Advantages of implementing this invention:

[0057] 1. Mapping accuracy and robustness are significantly improved.

[0058] Semantic information is deeply embedded in the entire SLAM chain: At the front end, point cloud registration residuals are assigned differentiated weights based on the static / dynamic attributes of semantic labels. Static features provide high-confidence constraints, while dynamic objects are effectively filtered out, fundamentally weakening the impact of dynamic obstacles on odometry drift. At the back end, three types of semantic constraints are introduced: semantic landmark observation factors, semantic geometric prior factors, and semantic co-visibility constraint factors. Among them, the information matrix of semantic landmark observation factors is adaptively weighted according to semantic category and confidence, enabling stable semantic landmarks to play a stronger anchoring role in optimization.

[0059] 2. Map update efficiency has been significantly improved.

[0060] This invention constructs a semantic landmark-level change detection mechanism, which performs joint association matching of spatial location and semantic category between newly acquired semantic landmarks and historical landmarks to accurately identify four types of change states: addition, removal, modification, and stability. Combined with the iSAM2 incremental smoothing solver, only the pose nodes and semantic landmark nodes within the affected Tile region are locally re-optimized, while the unchanged regions remain fixed.

[0061] 3. The map data structure is rich and can meet the needs of multiple modules.

[0062] The multi-layer high-precision map output by this invention contains five layers of heterogeneous data: point cloud layer, semantic landmark layer, vectorized road layer, semantic raster layer, and scene layer. It not only covers the dense geometric information required by the positioning module, but also provides the vectorized topology required by the planning module, the semantic raster prior required by the perception module, and the high-level scene semantic summary required by the decision reasoning module. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating the multi-sensor fusion semantic enhancement incremental autonomous driving mapping method described in this invention.

[0065] Figure 2 This is a schematic diagram illustrating the composition of the unified factor graph described in this invention;

[0066] Figure 3 This is a schematic diagram of the incremental map update and change detection process described in this invention;

[0067] Figure 4 This is a schematic diagram of the process of a multi-sensor fusion semantic enhancement incremental autonomous driving mapping system according to the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1

[0070] like Figure 1 As shown, a multi-sensor fusion semantic enhancement incremental autonomous driving mapping method includes:

[0071] Step S1: Acquire multi-sensor synchronization data, perform multimodal semantic perception and cross-modal fusion on the synchronization data, generate a unified semantic point cloud with semantic labels, and extract semantic landmarks based on the unified semantic point cloud;

[0072] After the vehicle initiates the mapping task, it first performs multi-sensor data acquisition and time synchronization. The system uses the GNSS PPS signal as the global time reference and employs the PTP / gPTP clock synchronization protocol to perform hardware-triggered synchronization of the LiDAR, surround-view cameras, and IMU, ensuring that the timestamps of data from all sensors are aligned to the same time reference, with time synchronization accuracy controlled within milliseconds. During data acquisition, the LiDAR scans the surrounding environment at a frequency of 10Hz and outputs a 3D point cloud. The surround-view cameras simultaneously acquire images of the vehicle's surroundings in multiple directions (front, rear, left, right, left-front, and right-front) at a frequency of 30Hz. RTK-GNSS provides centimeter-level global positioning information, and the IMU outputs the vehicle's three-axis acceleration and three-axis angular velocity information at a frequency of 100Hz.

[0073] After acquiring synchronized multi-sensor data, the system performs multimodal semantic perception and cross-modal fusion. At the vision end, the panoramic image is fed into a 2D multi-task neural network. This network uses a shared backbone to extract image features and then branches into four task heads, simultaneously outputting semantic segmentation results (distinguishing pixel-level semantic categories such as roads, vehicles, pedestrians, buildings, vegetation, and traffic signs), panoramic segmentation results (providing instance-level segmentation to distinguish different individual instances), object detection results (outputting 2D bounding boxes and categories of dynamic objects), and lane detection results (outputting pixel-level positions of lane lines). At the LiDAR end, the acquired LiDAR point cloud (as a specific form of 3D point cloud) is fed into a 3D point cloud semantic segmentation network (such as Cylinder3D), outputting a semantic label for each 3D point. Subsequently, the system utilizes a cross-modal attention mechanism to fuse 2D and 3D semantic features point-by-point: the semantic information in the 2D image is mapped to the 3D point cloud space through the camera projection matrix, and then the correlation between different modal features is calculated through attention weights to generate a unified semantic point cloud. The data structure of the unified semantic point cloud is P_unified = { (x, y, z, r, g, b, semantic_label, instance_id, confidence)}, where each point contains three-dimensional spatial coordinates, color information, semantic label, instance ID, and semantic confidence.

[0074] Based on a unified semantic point cloud, the system extracts semantic landmarks in 3D space. The semantic landmark extractor clusters and segments instances belonging to poles and signs in the unified semantic point cloud. For poles (such as streetlights, traffic sign poles, and utility poles), a cylindrical fitting method is used to reconstruct their 3D spatial position, radius, and orientation. For signs (such as traffic signs and street name signs), a planar fitting method is used to reconstruct their 3D spatial position, size, and normal direction. Finally, a structured semantic landmark set L = { L_j | j = 1, …, N} is formed. Each semantic landmark L_j contains a semantic category class_j, a spatial position P_j, geometric parameters G_j, and a unique instance identifier id_j.

[0075] Step S2: Based on the unified semantic point cloud and the semantic landmarks, perform semantic weighted front-end odometer calculation, assign different weights to the point cloud registration residuals according to the static / dynamic attributes of the semantic tags, and obtain the vehicle's initial pose estimation and keyframes.

[0076] After obtaining the unified semantic point cloud and semantic landmarks, the system performs semantically weighted front-end odometry. When performing scan-to-map registration between each newly scanned point cloud and the local map, the system assigns different weighting coefficients to the registration residual based on the object dynamic attributes represented by the semantic label of each point. The registration residual is defined as:

[0077] e_i = w(label_i) × e_geo_i

[0078] Where e_geo_i is the geometric residual of the i-th point (using a point-to-surface or point-to-plane distance metric), and w(label_i) is the weight coefficient corresponding to the semantic label label_i. The weight coefficient allocation strategy is as follows: point cloud registration residuals belonging to static land cover categories (such as roads, buildings, traffic signs, streetlights, and ground markings) are assigned high weights w_static (e.g., w_static = 1.0), point cloud registration residuals belonging to dynamic object categories (such as vehicles, pedestrians, and cyclists) are assigned zero weights or weights close to zero w_dynamic (e.g., w_dynamic = 0), and registration residuals belonging to intermediate categories (such as vegetation, guardrails, and shoulders) are assigned weights between the two w_intermediate (e.g., w_intermediate = 0.3~0.6). Through this weighted registration, static structures play a dominant role in odometry estimation, while dynamic objects are effectively filtered out.

[0079] Based on this, the system combines IMU pre-integration for state estimation. IMU pre-integration integrates acceleration and angular velocity measurements between adjacent keyframes to obtain relative motion constraints ΔR_ij, Δv_ij, Δp_ij, and their covariance matrix Σ_imu. The system fuses the relative pose transformation obtained from LiDAR weighted registration with the IMU pre-integration results using an IESKF (Iterative Error State Kalman Filter). The filter's state vector is x = [R, p, v, b_a, b_g]^T, where R is the rotation matrix, p is the translation vector, v is the velocity, and b_a and b_g are the biases of the accelerometer and gyroscope, respectively. The filter outputs the initial pose estimate of the vehicle at the current moment: x_k = [R_k, p_k].

[0080] The system simultaneously selects keyframes using a preset strategy—for example, inserting a new keyframe when the vehicle displacement exceeds 1 meter, the rotation angle exceeds 5 degrees, or the time interval exceeds 1 second—and performs data association matching between the semantic landmarks observed in the current keyframe and existing landmarks using the Hungarian algorithm. Specifically, a cost matrix C_{ij} = α × ||p_i - p_j|| + β × (class_i ≠ class_j) is constructed, where p_i and p_j are the spatial positions of the currently observed landmark and existing landmarks, class_i and class_j are their semantic categories, and α and β are weight coefficients. The correspondence between the same landmark in different keyframes is established by minimizing the total matching cost.

[0081] Step S3: Construct a unified factor graph for backend optimization. The unified factor graph includes at least a semantic landmark observation factor, a semantic geometric prior factor, and a semantic co-view constraint factor. The semantic landmark observation factor is used to constrain the reprojection error between the vehicle pose and the observed semantic landmarks. The semantic geometric prior factor is used to constrain the geometric properties of the semantic landmarks themselves. The semantic co-view constraint factor is used to constrain the relative pose relationship between different keyframes due to co-viewing the same semantic landmark.

[0082] After the system enters the backend optimization phase, a five-layer semantic scene graph is first constructed. This scene graph, from bottom to top, consists of:

[0083] (1) Sensor observation layer: Stores raw sensor data (images, point clouds, IMU data) and extracted semantic features, serving as the data foundation for the entire scene graph;

[0084] (2) Keyframe Pose Layer: Stores the pose nodes T_i ∈ SE(3) of each keyframe and their covariance information Σ_i;

[0085] (3) Semantic landmarks and structure layer: Stores the class_j, spatial location P_j, geometric parameters G_j of all semantic landmarks L_j, as well as the co-visibility relationship V_{ij} between keyframes and landmarks (indicating that semantic landmark j is observed by keyframe i);

[0086] (4) Road topology layer: Stores vectorized topological relationships of lane lines, road boundaries and intersections, including parameters such as lane connection relationships, road curvature, and lane width;

[0087] (5) Global region layer: Stores scene semantic summary information at the region level, such as region type (intersection, ramp, tunnel, parking lot, etc.), region semantic description and inter-region connection relationship.

[0088] The various levels are connected from the bottom up through the association of the underlying data, while the information from the higher levels can exert a priori guidance on the observations of the lower levels.

[0089] Based on a five-layer semantic scene graph, the system constructs a unified factor graph for backend optimization. The variable nodes of the factor graph include the vehicle poses X = {T_i | i = 1, …, M} for all keyframes, T_i = (R_i, t_i) ∈ SE(3), where R_i is the rotation matrix and t_i is the translation vector, and the spatial positions of all semantic landmarks L = {L_j | j = 1, …, N}, L_j ∈ ℝ³. For example... Figure 2 As shown, the factor nodes include at least seven types of factors: IMU pre-integration factors, relative motion factors, GPS prior factors, loop closure detection factors, semantic landmark observation factors, semantic geometric prior factors, and semantic co-visibility constraint factors. The objective function of the unified factor graph is:

[0090] minimize_{ X, L} {

[0091] ∑{(i,j) ∈ F_imu} || r_imu( T_i, T_j ) ||²{ Σ_imu}

[0092] ∑{(i,j) ∈ F_odo} || r_odo( T_i, T_j ) ||²{ Σ_odo}

[0093] ∑{i ∈ F_gps} || r_gps( T_i ) ||²{ ∑_gps}

[0094] ∑{(i,j) ∈ F_loop} || r_loop( T_i, T_j ) ||²{ Σ_loop}

[0095] ∑{(i,j) ∈ F_landmark} || r_landmark( T_i, L_j ) ||²{ Ω_ij}

[0096] ∑{j ∈ F_prior} || r_prior( L_j ) ||²{ ∑_prior}

[0097] ∑{(i,j,k) ∈ F_covis} || r_covis( T_i, T_j, L_k ) ||²{ Σ_covis}}

[0098] The meanings of each symbol are as follows:

[0099] F_imu is the set of IMU pre-integration factors, r_imu(T_i, T_j) is the IMU pre-integration residual, and Σ_imu is the IMU pre-integration covariance matrix; F_odo is the set of relative motion factors, r_odo(T_i, T_j) is the odometry relative motion residual, and Σ_odo is the odometry covariance matrix; F_gps is the set of GPS prior factors, r_gps(T_i) is the GPS observation residual, and Σ_gps is the GPS observation covariance matrix; F_loop is the set of loop closure detection factors, r_loop(T_i, T_j) is the loop closure detection residual, and Σ_loop is the loop closure detection covariance matrix; F_landmark is the set of semantic landmark observation factors, r_landmark(T_i, T_j) is the relative motion residual, and Σ_odo is the odometry covariance matrix; F_gps is the set of GPS prior factors, r_gps(T_i, T_j) is the relative motion residual, and Σ_gps ... L_j) represents the semantic landmark observation residuals, Ω_ij represents the adaptive information matrix of the semantic landmark observation factors; F_prior represents the set of semantic geometric prior factors, r_prior(L_j) represents the geometric prior residuals, and Σ_prior represents the geometric prior covariance matrix; F_covis represents the set of semantic covisibility constraint factors, r_covis(T_i, T_j, L_k) represents the semantic covisibility constraint residuals, and Σ_covis represents the covisibility constraint covariance matrix.

[0100] The above seven types of factors together constitute a unified optimization objective function. The system minimizes the global Mahalanobis distance of all residual terms by solving this objective function.

[0101] Step S4: Optimize the unified factor graph using an incremental smoothing solver, and output the optimized global pose trajectory and semantic map features;

[0102] After the factor graph is constructed, the system uses the iSAM2 incremental smoothing solver for optimization. iSAM2 is based on a Bayes tree structure; when adding new keyframes and semantic landmark observations, it only incrementally updates and re-linearizes the affected variable nodes in the Bayes tree, eliminating the need to resolve all historical variables each time. After multiple iterations and convergence, the system outputs the optimized global pose trajectory—containing the precise six-DOF pose of each keyframe in the world coordinate system—and the optimized semantic map elements—containing the category, spatial location, geometric parameters, and unique identifiers of all semantic landmarks.

[0103] Step S5: Compare the newly collected semantic landmarks with the historical semantic landmarks in the existing map, perform semantic landmark-level change detection, and identify the change status of the landmarks;

[0104] After acquiring several newly collected keyframes and their corresponding new semantic landmarks, the system performs semantic landmark-level change detection. Specifically, it performs joint spatial location and semantic category association matching between the newly collected semantic landmarks and historical semantic landmarks in the existing map. Based on the matching results, each semantic landmark is determined to be in one of four change states: newly added state, i.e., the newly collected landmark has no matching item in the existing map; removed state, i.e., the historical landmark in the existing map has no matching item in the newly collected data; modified state, i.e. the newly collected landmark successfully matches the historical landmark, but its spatial location or geometric parameters have changed; stable state, i.e. the newly collected landmark successfully matches the historical landmark and the difference in spatial location and geometric parameters is less than a preset threshold.

[0105] Step S6: Based on the change detection results, perform local re-optimization on the affected area, integrate the optimized map elements into the existing map in the form of incremental patches, and output the updated multi-layer high-precision map.

[0106] Based on the change detection results, the system performs incremental map updates. Specifically, based on the spatial location of semantic landmarks determined to be added, removed, or modified, the affected tile areas are identified. The system pre-divides the entire map space into fixed-size tile grids according to latitude, longitude, or Cartesian coordinates, with each tile containing map features within its coverage area. After identifying the affected tiles, the system only reconstructs the factor map for local optimization of the pose nodes and semantic landmark nodes within the tile area, while the pose nodes and semantic landmark nodes outside the tile area remain unchanged. After local optimization, the optimized map features are stitched together with the map features of the unaffected areas to generate an incremental patch.

[0107] Finally, the system outputs an updated multi-layered high-precision map, which comprises the following five layers: a point cloud layer, storing dense 3D point clouds; a semantic landmark layer, storing the category, spatial location, geometric parameters, and unique identifier of the semantic landmarks; a vectorized road layer, storing the vector geometric information of lane lines, road boundaries, and traffic signs; a semantic raster layer, which rasterizes and semantically labels the ground according to semantic categories; and a scene layer, storing high-level semantic information from the five semantic scene maps. These five layers of data together constitute a complete high-precision map, which is output in a standard format to in-vehicle or cloud-based map services for downstream autonomous driving modules such as positioning, planning, perception, and decision-making to access as needed.

[0108] Example 2

[0109] like Figure 3 As shown, this embodiment takes an autonomous driving data collection vehicle performing a new round of data collection tasks in an area where the initial mapping has been completed as an example to describe in detail how the present invention uses newly collected multi-sensor data to incrementally update the existing high-precision map, including the complete process of map positioning, semantic landmark level change detection, incremental factor map update, tile local re-optimization, and incremental patch generation and fusion.

[0110] Step A1: Locate the newly acquired Session on an existing map;

[0111] After the vehicle enters the mapped area, the system first performs map localization. Specifically, the vehicle acquires the current LiDAR point cloud and surround view image, which are then processed by the Layer 2 multimodal semantic perception module to generate a unified semantic point cloud and extract semantic landmarks. The system matches the extracted semantic landmarks with those stored in the existing map, using ICP registration combined with semantic label consistency verification to calculate the initial pose estimate of the current vehicle in the existing map coordinate system. For areas where mapping is completed for the first time, the map already contains a high-precision semantic landmark layer (L2), where each semantic landmark has a unique identifier and precise spatial location. Therefore, map localization can be completed within seconds, with a positioning accuracy better than 30cm.

[0112] Step A2: Run the front-end odometer to generate a new trajectory and new road signs;

[0113] After completing map localization, the system uses the localization result as the initial value and performs semantically weighted front-end odometry under the constraints of the existing map. Consistent with the process described in step two of Example 1, the system performs semantically weighted Scan-to-Map registration based on a unified semantic point cloud and semantic landmarks. It assigns differentiated weights to the point cloud registration residuals according to the static / dynamic attributes of the semantic labels, and obtains vehicle pose estimation and keyframes through IESKF (Iterative Error State Kalman Filter) using IMU pre-integration. During this process, the system maintains real-time association between newly acquired semantic landmarks and historical semantic landmarks in the existing map, establishing a data correspondence. After front-end processing is complete, the system outputs the local trajectory X_new = {T_i | i = 1, …,P} for the new session and the newly observed semantic landmarks L_new = {L_j | j = 1, …, Q}.

[0114] Step A3: Perform semantic landmark-level change detection;

[0115] The system performs joint spatial location and semantic category matching between the newly acquired semantic landmark L_new and the historical semantic landmark L_old in the existing map. The specific process is as follows:

[0116] For each newly acquired semantic landmark L_j ∈ L_new, the system searches the existing map for historical landmark candidates with the same semantic category as L_j, using L_j's spatial location as the center and a preset radius (e.g., 3.0 meters) as the search range. If no match is found within the search range, L_j is determined to be in an added state and added to the list of new landmarks.

[0117] If one or more matching candidates exist within the search range, the system calculates the spatial distance between L_j and each candidate landmark, selecting the candidate with the smallest distance as the best match. If the minimum spatial distance is less than the position change threshold (e.g., 0.5 meters), the landmark is determined to be in a stable state, and its correspondence with historical landmarks is recorded; if the minimum spatial distance is greater than or equal to the position change threshold, the landmark is determined to be in a modified state, added to the modified landmark list, and the corresponding historical landmark ID is recorded.

[0118] In addition, the system also needs to detect historical landmarks that have been removed. Specifically, the system obtains all historical semantic landmarks within the visible range of the current new trajectory. If a historical landmark does not form a stable or modified match with any newly acquired landmark during the above matching process, and its area is effectively covered by the new trajectory, then the historical landmark is determined to be removed and added to the removed landmark list.

[0119] After the change detection is completed, the system outputs four types of change results: a list of newly added landmarks (L_added), a list of removed landmark IDs (L_removed), a list of modified landmark pairs (L_modified, which includes the correspondence between historical landmarks and new landmarks and the amount of change), and a list of stable landmark pairs (L_stable, which includes the correspondence between historical landmarks and new landmarks).

[0120] Step A4: Update the incremental factor graph;

[0121] Based on the change detection results, the system performs incremental factor graph updates. The core idea of ​​this step is to fix stable landmarks as global anchor points and only perform incremental factor graph optimization on landmarks that have changed and their associated keyframes.

[0122] Specifically, for semantic landmarks determined to be in a stable state, the system adds their spatial location as a fixed prior anchor point to the factor graph, assigns a high weight (e.g., a weight coefficient of 100.0) to the prior factor information matrix corresponding to the landmark location, and keeps it unchanged in subsequent optimizations. For semantic landmarks determined to be in a newly added state, the system adds a corresponding landmark variable node L_added to the factor graph and establishes a semantic landmark observation factor between the keyframe pose node of the landmark observed in the new session and the newly added landmark node. For semantic landmarks determined to be in a modified state, the system locates the variable node corresponding to the historical landmark in the factor graph, updates its observation value, and adds the semantic landmark observation factor corresponding to the new observation to the factor graph. In the inference chain, for semantic landmarks determined to be in a removal state, the system marks the landmark node as to be removed in the factor graph, but does not delete the node in this round of optimization. Instead, it sets the prior factor information matrix corresponding to the landmark node to a low weight (e.g., a weight coefficient of 0.01), so that its impact on the overall objective function in the optimization approaches zero. It is then removed from the Bayes tree after the local re-optimization is completed.

[0123] The system simultaneously adds the newly added keyframe pose nodes in the new session to the factor map and establishes a loop closure detection factor (if a loop closure relationship exists) between the new keyframe and stable historical keyframes in the existing map. For semantic landmarks determined to be stable, since they have already been globally optimized in the existing map, the system does not add new observation factors for them. Instead, it uses the optimized position of the landmark in the existing map as a strong prior anchor point, which is only used to constrain the pose of the new keyframe.

[0124] Step A5: Local re-optimization of affected tiles;

[0125] After the incremental factor map is updated, the system performs local re-optimization. Specifically, the system determines the affected tile area based on the spatial location of semantic landmarks that are determined to be added, removed, or modified. The system pre-divides the entire map space into a fixed-size tile grid according to Cartesian coordinates (e.g., each tile is 100 meters × 100 meters in size). Each tile is uniquely identified by its geographic index coordinates (id_x, id_y) and contains all map features within that coverage area.

[0126] The system collects the location coordinates of all changed landmarks (added, modified, and removed), calculates their respective tile indices, and forms the affected tile set T_affected = {Tile(P_j) | state(P_j) ∈ {added, removed, modified}}. After identifying the affected tiles, the system only reconstructs the factor graph for local optimization of the pose nodes and semantic landmark nodes within T_affected. Specifically, it extracts all keyframe pose nodes, semantic landmark nodes, and factors directly associated with these nodes (IMU pre-integration factor, relative motion factor, GPS prior factor, loop closure detection factor, semantic landmark observation factor, semantic geometric prior factor, and semantic co-visibility constraint factor) within the affected tile, constructs a local factor graph, and performs nonlinear least squares optimization. Pose nodes and semantic landmark nodes outside the tile region remain fixed and participate as constants in the residual calculation in the objective function but do not participate in the optimization iteration.

[0127] Local optimization employs the Levenberg-Marquardt solver, with the optimization variables being the keyframe poses and semantic landmark positions within the affected tiles. Since the number of nodes in a local region is much smaller than the global scale, optimization can be completed within seconds. After local optimization converges, the system stitches the optimized map features with those from the unaffected areas.

[0128] Step A6: Incremental Patch Generation and Map Version Management;

[0129] After local optimization is complete, the system generates an incremental patch. The incremental patch contains the following information:

[0130] Add a new list of road signs and their optimized spatial location, semantic category, geometric parameters, and unique identifiers;

[0131] List of historical road sign IDs that need to be removed;

[0132] Modify the ID of the road sign and its updated spatial location and geometric parameters;

[0133] The updated keyframe pose trajectory within the affected tile (used to support subsequent localization and data association).

[0134] Incremental patches are serialized in binary or JSON format. The system submits the incremental patch to the cloud map service, which verifies the patch (checking data integrity, semantic consistency, and topological compatibility with existing maps) before merging it into the main map database to generate a new version map (e.g., v1.0 → v1.1). The map version number corresponds one-to-one with the incremental patch, supporting version rollback and difference tracking.

[0135] In multi-vehicle crowdsourced fusion scenarios, data collected by multiple data collection vehicles at different times generates their own incremental patches, which are then uploaded to the cloud. The cloud service sorts the incremental patches by timestamp and performs a weighted average fusion (information matrix weighting) on ​​observations from different vehicles within the same tile area. The accuracy of the fused road sign locations continuously improves with the increase in the number of observations. After fusion is complete, the cloud generates a unified version of the new map and distributes it to the fleet.

[0136] In this embodiment, based on semantic landmark-level change detection and the iSAM2 incremental optimization mechanism, a single session incremental update can be completed within 5 minutes. Moreover, as the map size increases, the increase in update time is much smaller than that of traditional global optimization schemes. The strategy of using stable landmarks as anchor points ensures that the accuracy of unchanged areas in the existing map is not affected by the introduction of new data, thus realizing near real-time crowdsourced maintenance of high-precision maps.

[0137] Example 3

[0138] This embodiment uses an autonomous vehicle performing a mapping task in a complex scenario as an example to describe in detail how the present invention adaptively switches between fast thinking and slow thinking modes based on scenario complexity, and how, in slow thinking mode, it generates structured semantic reasoning results through an inference chain, and combines the GRPO reward model with world model imagination training to achieve self-iterative optimization of system performance. This embodiment enhances the upper-level decision-making capabilities of the main mapping system and is suitable for challenging scenarios such as geometric degradation, dynamic density, or drastic changes in lighting.

[0139] Step B1: Calculate the scene complexity metrics for the current scene;

[0140] During the vehicle's mapping task, the system calculates a quantifiable metric for the current scene's complexity in real time. This metric is used to quantitatively assess the challenge posed by the current scene to pure geometric SLAM methods, and includes calculations across at least the following four dimensions:

[0141] (1) Point cloud feature entropy: Extract features (edge ​​points and planar points) from the current frame LiDAR point cloud and count the uniformity of feature point distribution in the point cloud. Divide the point cloud space into several voxel grids and calculate the distribution entropy value of feature points in each grid. The higher the feature entropy, the richer the scene geometry. The lower the entropy value, the more the scene tends to be geometrically degraded (such as long straight tunnels and open open areas).

[0142] (2) The proportion of dynamic objects in the total number of point clouds: The proportion of the number of points in the current frame that are marked as dynamic objects (vehicles, pedestrians, cyclists, etc.) is calculated based on the semantic segmentation results. The higher the proportion, the more serious the dynamic interference.

[0143] (3) RTK-GNSS signal confidence: The GNSS confidence is comprehensively evaluated based on parameters such as the fixed solution state of the RTK-GNSS receiver, the number of satellites, and the position accuracy factor (PDOP). The lower the confidence, the less reliable the global positioning prior is.

[0144] (4) Image texture richness: Perform gray-level co-occurrence matrix or FAST corner detection on the panoramic image, and count the number and distribution of corners in the image. The lower the texture richness, the more visual features are lacking (such as white tunnel walls and empty road surfaces).

[0145] Step B2: Dual-mode gating decision;

[0146] The system weights and fuses the above four dimensions of indicators to obtain a comprehensive scenario complexity score, S_complexity. Based on this score, the system uses a dual-state hysteresis gating decision-maker to determine whether to adopt a fast thinking mode or a slow thinking mode.

[0147] When S_complexity is below a first threshold (e.g., S_complexity < 0.3), the system determines the current scene to be a simple scene and switches to fast thinking mode. In fast thinking mode, the system only performs pose tracking using geometric feature-based VIO / LIO odometry and does not trigger the semantic reasoning enhancement module. This mode has low computational overhead, meets real-time requirements, and is suitable for conventional scenes with rich geometry, few dynamic objects, and good GNSS signal.

[0148] When S_complexity exceeds the second threshold (e.g., S_complexity > 0.7), the system determines the current scenario to be complex and switches to slow thinking mode. In slow thinking mode, the system triggers complete semantic reasoning enhancement, including calling the reasoning chain to generate structured semantic reasoning results for the current scenario, and adjusting the weight allocation strategy of the semantically weighted front-end odometer and the information matrix configuration of each semantic factor in the unified factor graph based on the reasoning results.

[0149] The first threshold and the second threshold form a hysteresis interval (0.3 ≤ S_complexity ≤ 0.7). When the scene complexity falls into this interval, the system maintains the current mode to avoid frequent mode switching caused by small fluctuations in the complexity index.

[0150] Step B3: Generation of reasoning chains in slow thinking mode;

[0151] When the system switches to slow thinking mode, it invokes the inference chain module to generate structured semantic reasoning results for the current scene. The inference chain is based on a large language model or a multimodal large model, combining the semantic perception results of the current frame (semantic point cloud, semantic landmarks, 2D detection results) and scene context information to generate structured text output according to a preset inference path template. In this embodiment, one implementation of the inference chain is RP-CoT (Reasoning Path-Chain of Thought).

[0152] The inference chain follows the following inference path template: First, scene identification is performed to determine the current scene type (e.g., tunnel exit, congested intersection, spiral passage in underground parking garage, etc.); then, degradation analysis is performed to identify key factors that cause pure geometric SLAM to fail (e.g., sparse geometric features, dynamic object occlusion, drastic changes in lighting, etc.); next, semantic constraint recommendation is performed, recommending the types of semantic factors that should be activated first based on the scene type (e.g., in a tunnel, it is recommended to activate the vertical constraint among the semantic landmark observation factor and geometric prior factor, and in a congested intersection, it is recommended to activate the semantic co-visibility constraint factor); finally, weight adjustment suggestions are made, recommending specific adjustment values ​​for the semantic class weight α (class) and confidence weight β (confidence) for the current scene.

[0153] For example, in a tunnel exit scenario, the inference chain outputs the following inference result: the scene type is "severe exposure overexposure + narrow and long geometric degradation"; the degradation factors are "low point cloud feature entropy (long straight tunnel), extremely low image texture richness (white tunnel wall), and drastic change in illumination intensity from dark to bright"; the semantic constraint recommendation is "prioritize activating traffic sign constraints and lamppost verticality constraints in the semantic road sign observation factors"; the weight adjustment suggestion is "traffic sign α=1.0, lamppost α=0.9, dynamic object α=0.0, and ground point registration weight reduced to 0.3". Based on this inference result, the system automatically adjusts the front-end odometer weight allocation and the back-end factor graph information matrix configuration.

[0154] Step B4: GRPO reward model optimization;

[0155] The system employs a GRPO (Group Relative Policy Optimization) reward model to perform multi-dimensional evaluation and optimization of inference results and mapping quality. The reward model calculates the comprehensive reward value based on the following five dimensions:

[0156] (1) Positioning accuracy reward: Based on the ground truth (RTK-GNSS) or closed-loop detection error, calculate the absolute pose error (ATE) of the current session. The smaller the positioning error, the higher the reward.

[0157] (2) Reprojection error reward: Calculate the average reprojection residual of all semantic landmark observation factors in the factor graph. The smaller the residual, the better the semantic constraint consistency and the higher the reward.

[0158] (3) Semantic consistency reward: detect the consistency of semantic labels of the same semantic landmark in different keyframes, as well as the degree of conformity between the semantic landmark and geometric priors (such as verticality and orientation). The higher the consistency, the higher the reward.

[0159] (4) Calculation time penalty: Record the extra calculation time in slow thinking mode (including RP-CoT inference, factor graph incremental update, etc.), and give a negative reward for the time-out part.

[0160] (5) Loopback Recall Reward: The higher the loopback detection recall rate of the detection system in complex scenarios, the higher the reward.

[0161] The reward values ​​from the five dimensions mentioned above are summed according to preset weights to obtain the comprehensive reward value R_total. This comprehensive reward value serves as a feedback signal, used in subsequent iterations to adjust the RP-CoT inference strategy, semantic weight configuration, and gating decision threshold.

[0162] Step B5: World Model Visualization Training;

[0163] To further enhance the system's prediction and decision-making capabilities in complex scenarios, a world model is introduced for imaginative training. The world model learns the spatiotemporal evolution of the environment in latent space, enabling it to predict sensor observations and vehicle states at future points in time based on current observations.

[0164] In this embodiment, the world model adopts an architecture based on variational autoencoder or diffusion model, encoding the semantic point cloud, semantic landmarks, and vehicle pose of the current frame into latent state vectors, predicting the latent state at time t+1 in the latent space, and then decoding it into predicted observations. The system compares the prediction results of the world model with the actual observations, calculates the prediction error, and uses this error as an unsupervised training signal to update the world model parameters.

[0165] The imaginative training employs a "fast forced single-step rollout" strategy—at each new keyframe, the system forces the world model to perform a single-step forward prediction (even if it is currently in fast-thinking mode), comparing the prediction results with the actual observations, and continuously accumulating training signals without increasing the latency of downstream tasks. Comparative data shows that fast forced single-step rollout achieves more than 85% improvement in prediction accuracy with only 20% of the computational cost of conventional multi-step rollout, achieving a good balance between efficiency and performance.

[0166] The predictive capabilities of the world model can be used in the following scenarios: when GNSS signals are briefly lost, the world model predicts the current pose based on historical observations to help the odometer get through the signal loss period; when dynamic objects occlude semantic landmarks, the world model predicts the current position of the occluded landmarks to maintain the continuity of semantic constraints.

[0167] Step B6: Self-iterative loop closure and performance convergence;

[0168] The system continuously optimizes its performance through a self-iterative closed loop of "data acquisition → model update → performance evaluation → strategy adjustment". After each session, the system uploads the collected data, inference results, mapping quality indicators (localization error, reprojection error, loop closure recall, etc.), and GRPO comprehensive reward value to the cloud training platform. The cloud platform accumulates data from multiple sessions and periodically performs joint optimization on the RP-CoT inference model, semantic weight allocation strategy, and gating decision threshold, generating updated model parameters and strategy configurations, which are then distributed to all vehicles in the fleet.

[0169] After several rounds of iteration, the system's main performance indicators gradually improved and stabilized: absolute mapping accuracy gradually converged to within 10cm (when RTK was available), loop closure detection recall converged to over 90%, incremental update latency converged to within 5 minutes, and gating decision accuracy (i.e., the probability of selecting the correct mode) converged to over 95%. Once the plateau was reached, the system entered a stable operation phase, adaptively allocating computing resources while ensuring mapping accuracy, providing reliable support for large-scale commercial deployment.

[0170] In this embodiment, the introduction of a dual-mode gating decision-making mechanism enables the system to achieve an optimal balance between slow thinking with high computational cost and fast thinking with low latency. The inference chain provides a scenario-adaptive reasoning basis for configuring semantic constraints, the GRPO reward model ensures the quantifiable and evaluable nature of the optimization objective, and world model imagination training enhances the system's robustness in extreme scenarios. These mechanisms collectively improve the mapping accuracy and robustness of this invention in challenging scenarios such as geometric degradation, dynamic density, and drastic lighting changes.

[0171] Example 4

[0172] A multi-sensor fusion semantic enhancement incremental autonomous driving mapping system includes:

[0173] The acquisition module M1 acquires multi-sensor synchronous data, performs multimodal semantic perception and cross-modal fusion on the synchronous data, generates a unified semantic point cloud with semantic labels, and extracts semantic landmarks based on the unified semantic point cloud.

[0174] The weighting module M2, based on the unified semantic point cloud and the semantic landmarks, performs semantic weighted front-end odometer measurement, assigns different weights to the point cloud registration residuals according to the static / dynamic attributes of the semantic tags, and obtains the vehicle's initial pose estimation and keyframes.

[0175] The optimization module M3 constructs a unified factor graph for backend optimization. The unified factor graph includes at least a semantic landmark observation factor, a semantic geometric prior factor, and a semantic co-view constraint factor. The semantic landmark observation factor is used to constrain the reprojection error between the vehicle pose and the observed semantic landmarks. The semantic geometric prior factor is used to constrain the geometric properties of the semantic landmarks themselves. The semantic co-view constraint factor is used to constrain the relative pose relationship between different keyframes due to co-viewing the same semantic landmark.

[0176] The map module M4 uses an incremental smoothing solver to optimize the unified factor map and outputs the optimized global pose trajectory and semantic map features.

[0177] The comparison module M5 compares the newly acquired semantic landmarks with the historical semantic landmarks in the existing map, performs semantic landmark-level change detection, and identifies the change status of the landmarks.

[0178] The patch module M6, based on the change detection results, performs local re-optimization on the affected area, integrates the optimized map elements into the existing map in the form of incremental patches, and outputs an updated multi-layer high-precision map.

[0179] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-sensor fusion semantic enhancement incremental autonomous driving mapping method, characterized in that, include: Acquire multi-sensor synchronous data, perform multimodal semantic perception and cross-modal fusion on the synchronous data, generate a unified semantic point cloud with semantic labels, and extract semantic landmarks based on the unified semantic point cloud; Based on the unified semantic point cloud and the semantic landmarks, a semantically weighted front-end odometer is executed. Different weights are assigned to the point cloud registration residuals according to the static / dynamic attributes of the semantic tags to obtain the vehicle's initial pose estimation and keyframes. A unified factor graph is constructed for backend optimization. The unified factor graph includes at least semantic landmark observation factors, semantic geometric prior factors, and semantic co-visibility constraint factors. An incremental smoothing solver is used to optimize the unified factor graph, and the optimized global pose trajectory and semantic map features are output. The newly acquired semantic landmarks are compared and correlated with historical semantic landmarks in the existing map to perform semantic landmark-level change detection and identify the change status of the landmarks. Based on the change detection results, the affected areas are locally re-optimized, and the optimized map elements are integrated into the existing map in the form of incremental patches to output an updated multi-layer high-precision map.

2. The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method according to claim 1, characterized in that, The multimodal semantic perception and cross-modal fusion specifically include: A 2D multi-task neural network is used to perform semantic segmentation, panoramic segmentation, object detection, and lane line detection on the surround view image to obtain 2D semantic features; a 3D point cloud semantic segmentation network is used to perform point-by-point semantic segmentation on the LiDAR point cloud to obtain 3D semantic features; and a cross-modal attention mechanism is used to fuse the 2D semantic features and the 3D semantic features point-by-point to generate the unified semantic point cloud. The extraction of semantic landmarks includes: clustering instances of rod-shaped objects and signs in the unified semantic point cloud, and reconstructing their three-dimensional spatial position and geometric parameters using cylindrical fitting or planar fitting methods.

3. The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method according to claim 1, characterized in that, In the semantically weighted front-end odometry, different weights are assigned to the point cloud registration residuals based on the static / dynamic attributes of the semantic labels, specifically including: Semantic labels are classified into static high-weight categories, dynamic zero-weight categories, and intermediate-weight categories. Point cloud registration residuals belonging to static land cover categories are assigned a first weight, and point cloud registration residuals belonging to dynamic object categories are assigned a second weight. The first weight is greater than the second weight. The second weight includes zero weight and weights close to zero.

4. The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method according to claim 1, characterized in that, The unified factor graph includes at least semantic landmark observation factors, semantic geometric prior factors, and semantic co-visibility constraint factors, wherein: The semantic landmark observation factor is constructed based on the reprojection residual between the vehicle pose and the spatial location of the semantic landmark. The information matrix of the residual is adaptively weighted according to the semantic category and semantic confidence. The semantic category includes static land feature category and dynamic object category, and different semantic categories correspond to different weight values. The semantic geometry prior factors include at least one of the following: the verticality constraint factor of the pole-shaped semantic landmark, the orientation constraint factor of the sign semantic landmark, and the parallel constraint factor between landmarks of the same semantic category. The semantic co-view constraint factor is derived from the co-view observation of the same semantic landmark by multiple consecutive keyframes and is used to constrain the relative pose between the multiple keyframes.

5. The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method according to claim 1, characterized in that, It also includes constructing a five-layer semantic scene graph, which, from bottom to top, consists of: sensor observation layer, keyframe pose layer, semantic landmark and structure layer, road topology layer, and global region layer; The sensor observation layer stores raw sensor data and extracted semantic features; The keyframe pose layer stores the optimized keyframe pose and its covariance; the semantic landmark and structure layer stores the category, position, geometric parameters and co-view relationship of the semantic landmark; the road topology layer stores the vectorized topological relationship of lane lines, road boundaries and intersections; and the global region layer stores region-level scene semantic summary information. The construction of the unified factor graph is based on the relationship between each level in the five-layer semantic scene graph, establishing constraint edges between factor nodes and variable nodes.

6. The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method according to claim 5, characterized in that, The multi-layer high-precision map includes the following five layers: Point cloud layer, storing 3D point clouds; The semantic landmark layer stores the category, spatial location, geometric parameters, and unique identifier of the semantic landmarks; The vectorized road layer stores the vector geometric information of lane lines, road boundaries, and traffic signs; Semantic raster layer: The ground is rasterized and semantically labeled according to semantic categories; The scene layer stores high-level semantic information from the five-layer semantic scene graph.

7. The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method according to claim 1, characterized in that, The semantic landmark-level change detection specifically includes: The newly acquired semantic landmarks are matched with historical semantic landmarks in the existing map in terms of spatial location and semantic category. Based on the matching results, each semantic landmark is determined to be one of the following four change states: A new status has been added: newly collected road signs have no matching items in the existing map; Removed status; historical road signs in the existing map have no matching entries in the newly collected data. The status has been modified; the newly collected landmark successfully matches the historical landmark, but its spatial location or geometric parameters have changed. In a stable state, newly collected road signs are successfully matched with historical road signs, and the differences in spatial location and geometric parameters are less than the preset threshold.

8. The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method according to claim 7, characterized in that, The local re-optimization of the affected area based on the change detection results specifically includes: The affected area is determined based on the spatial location of the semantic landmarks that are identified as new, removed, or modified. The factor graph is reconstructed and locally optimized only for the pose nodes and semantic landmark nodes within the affected region, while the pose nodes and semantic landmark nodes outside the affected region remain unchanged. The locally optimized map features are stitched together with map features from unaffected areas to generate an incremental patch.

9. The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method according to claim 1, characterized in that, Before performing semantically weighted front-end odometer calculation, a dual-mode gating decision-making step based on scenario complexity is also included: Calculate the scene complexity metrics for the current scene, wherein the scene complexity metrics include at least one of the following: point cloud feature entropy, the proportion of dynamic objects in the total number of point clouds, RTK-GNSS signal confidence, and image texture richness. When the complexity of the scenario is below the first threshold, switch to fast thinking mode and only perform pose tracking using VIO / LIO odometry based on geometric features; When the complexity of the scenario exceeds the second threshold, the system switches to slow thinking mode and triggers semantic reasoning enhancement. The semantic reasoning enhancement includes calling the reasoning chain to generate a structured semantic reasoning result for the current scenario, and adjusting the weight allocation strategy of the semantically weighted front-end odometer and the information matrix configuration of each semantic factor in the unified factor graph based on the reasoning result. The first threshold is lower than the second threshold, and the two form a hysteresis interval to avoid frequent mode switching.

10. A multi-sensor fusion semantically enhanced incremental autonomous driving mapping system, characterized in that, The multi-sensor fusion semantic enhancement incremental autonomous driving mapping method based on any one of claims 1 to 9 includes: The acquisition module acquires multi-sensor synchronous data, performs multimodal semantic perception and cross-modal fusion on the synchronous data, generates a unified semantic point cloud with semantic labels, and extracts semantic landmarks based on the unified semantic point cloud. The weighting module, based on the unified semantic point cloud and the semantic landmarks, performs semantic weighted front-end odometer measurement, assigns different weights to the point cloud registration residuals according to the static / dynamic attributes of the semantic tags, and obtains the vehicle's initial pose estimation and keyframes. The optimization module constructs a unified factor graph for backend optimization. The unified factor graph includes at least semantic landmark observation factors, semantic geometric prior factors, and semantic co-view constraint factors. The semantic landmark observation factors are used to constrain the reprojection error between the vehicle pose and the semantic landmark. The semantic geometric prior factors are used to constrain the geometric properties of the semantic landmark. The semantic co-view constraint factors are used to constrain the relative pose relationship between multiple keyframes that co-view the same semantic landmark. The map module uses an incremental smoothing solver to optimize the unified factor map and outputs the optimized global pose trajectory and semantic map features. The comparison module compares the newly acquired semantic landmarks with the historical semantic landmarks in the existing map, performs semantic landmark-level change detection, and identifies the change status of the landmarks. The patch module, based on the change detection results, performs local re-optimization on the affected areas, integrates the optimized map elements into the existing map in the form of incremental patches, and outputs an updated multi-layer high-precision map.

Images