A deep learning-based adaptive mapping and positioning method, system and medium

Abstract

The application discloses a kind of adaptive mapping and positioning method, system and medium based on deep learning, it is related to unmanned vehicle;3D point cloud data after motion compensation is projected as 2D laser image, and multiple channel feature image is obtained;According to multiple channel feature image, dynamic object detection is carried out using neural network based on ResNet50 architecture, and the dynamic object probability value of each scanning point is obtained;Initial grid map is constructed based on 3D point cloud data after motion compensation;The laser beam hit frequency alpha and laser beam pass frequency beta of each grid cell are updated using dynamic object probability value;According to the laser beam hit frequency alpha and laser beam pass frequency beta after updating, the reflection probability of each grid cell is calculated;According to reflection probability, grid cell in initial grid map is screened for dynamic and static object;Adaptive weighted fusion is carried out to pose prediction value and pose observation value using Kalman filtering algorithm, and final positioning result is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned driving, and in particular to a self-adaptive mapping and positioning method and system based on deep learning and a medium. BACKGROUND

[0002] With the rapid development of autonomous robot systems and autonomous driving technologies, high-precision simultaneous localization and mapping (SLAM) technology has become one of the key technologies to realize autonomous navigation. 3D laser radar (LiDAR) has become the core sensor for environment perception and map construction of autonomous robots due to its ability to provide high-precision and high-density three-dimensional point cloud data. In a static environment, a SLAM system based on laser radar can achieve centimeter-level positioning accuracy through a scanning matching algorithm. However, as the application scenarios expand from structured static environments to complex dynamic environments such as city streets, shopping malls, and airports, traditional laser radar SLAM systems face severe challenges. In these scenarios, the large number of dynamic objects such as pedestrians, vehicles, and shopping carts seriously affects the accuracy of laser radar scanning matching, resulting in a significant decrease in positioning accuracy.

[0003] Specific problems existing in the prior art:

[0004] Interference of dynamic objects on laser scanning matching: Traditional laser radar positioning methods mainly rely on matching the current scan with the constructed map or the previous scan. In dynamic object dense areas, these methods have the following problems: (1) Mis-matching problem: the position change of dynamic objects at different times leads to inconsistent point cloud features, and the scanning matching algorithm is easy to mis-match the point cloud of dynamic objects to the wrong position, resulting in a large positioning error. (2) Improper weight distribution: existing scanning matching algorithms assign the same weight to all point clouds, which cannot distinguish between static environment points and dynamic object points, resulting in systematic bias when dynamic object points participate in matching calculation. (3) Map pollution problem: the trajectory of dynamic objects is incorrectly integrated into the static map, forming a ghost phenomenon, which not only affects the quality of the map, but also produces cumulative errors in subsequent positioning.

[0005] Limitations of existing dynamic object detection methods: (1) Geometric feature-based methods: rely on multi-frame point cloud difference comparison or motion consistency detection, with high computational complexity, difficult to process in real time, and poor detection effect for objects that move slowly or are stationary and then move. (2) Motion model-based methods: require pre-definition of object motion patterns, making it difficult to adapt to diverse dynamic object types and motion patterns in complex scenarios. (3) Insufficient detection accuracy: existing methods mostly use object-level or cluster-level detection, which cannot achieve fine-grained dynamic probability estimation at the point level, resulting in insufficient accuracy when dealing with partially occluded or boundary ambiguous dynamic objects.

[0006] Adaptability loss in multi-sensor fusion: traditional multi-sensor fusion positioning methods (such as Kalman filter-based fusion) have the following shortcomings: (1) fixed fusion weight: the measurement noise covariance matrix is usually set to a fixed value, which cannot be adaptively adjusted according to the dynamic degree of the environment, and still gives a higher weight to the laser matching result in the dynamic object dense area. (2) Lack of environmental awareness: the fusion algorithm cannot perceive the dynamic object distribution of the current environment, and cannot adjust the reliability of sensor data according to the characteristics of the scene. (3) Lack of robustness: in a high dynamic environment, the fixed weight fusion strategy is easily affected by laser matching errors, resulting in a sharp decline in overall positioning performance.

[0007] In summary, the prior art lacks an effective dynamic object identification mechanism and an adaptive fusion strategy when dealing with laser radar SLAM problems in dynamic environments, and cannot fully suppress the interference of dynamic objects on laser scan matching, making it difficult to maintain high-precision positioning performance in complex dynamic scenes. Therefore, there is an urgent need for a new method that can accurately identify dynamic objects and adaptively adjust the fusion weight to improve positioning accuracy and robustness in dynamic environments. SUMMARY

[0008] In view of the fact that laser radar scan matching in dynamic object dense areas is easily disturbed, resulting in a decline in positioning accuracy, the present application provides an adaptive mapping and positioning method, system and medium based on deep learning, which realizes pixel-level dynamic object probability estimation through a deep learning algorithm, and adaptively adjusts the multi-sensor fusion weight according to the dynamic object probability, reduces the influence of laser matching in a high dynamic environment, and further improves the positioning accuracy in complex motion scenes.

[0009] One aspect of the present application provides an adaptive mapping and positioning method based on deep learning, comprising: collecting multi-sensor raw data, the multi-sensor raw data including laser radar 3D point cloud data, IMU inertial measurement data and odometer pose data; time synchronizing the multi-sensor raw data to obtain time-aligned sensor data; based on the odometer pose data in the time-aligned sensor data, performing motion compensation on the laser radar 3D point cloud data to obtain motion-compensated 3D point cloud data;

[0010] The motion-compensated 3D point cloud data is projected into a 2D laser image to obtain a multi-channel feature image including distance, intensity, height, and distance difference. Based on the multi-channel feature image, a neural network based on the ResNet50 architecture is used for dynamic object detection to obtain the dynamic object probability value for each scan point. An initial mesh map is constructed based on the motion-compensated 3D point cloud data. The laser beam hit count α and laser beam pass count β of each mesh cell are weighted and updated using the dynamic object probability value. The reflection probability of each mesh cell is calculated based on the updated laser beam hit count α and laser beam pass count β. The mesh cells in the initial mesh map are then filtered for dynamic and static objects based on the reflection probability to obtain a static 3D mesh map with dynamic objects removed.

[0011] The pose prediction is obtained by estimating the trajectory based on time-aligned sensor data; the pose observation is obtained by performing NDT scanning matching based on a static 3D mesh map; the pose prediction and pose observation are adaptively weighted and fused using a Kalman filter algorithm to obtain the final localization result; among these, the measurement noise covariance matrix of the Kalman filter algorithm is adjusted according to the probability value of dynamic objects.

[0012] Furthermore, the motion-compensated 3D point cloud data is obtained, including: acquiring 3D point cloud data from the LiDAR and odometry pose data, and acquiring the start time of a single-frame scan by the LiDAR. and end time and timestamps of each scan point Store the odometry pose data in chronological order to construct a pose temporal queue; based on the pose temporal queue, analyze the LiDAR scanning time periods. Piecewise linear interpolation is performed on the poses of the scan points within the area; based on the interpolated poses at each time step, coordinate transformation is used to convert the poses at each time step... LiDAR scanning points Switch to the start time In the coordinate system, we obtain motion-compensated 3D point cloud data.

[0013] Specifically, during the LiDAR scanning process (typically requiring 100ms to complete a 360-degree scan), the robot itself is in motion. Without motion compensation, the same static object will appear in different positions at the start and end of the scan, creating a ghosting effect. This point cloud distortion caused by self-motion can be misidentified as dynamic features, severely interfering with the deep learning model's judgment of real dynamic objects. Through precise timestamp-level motion compensation, all scanned points are unified to the same coordinate system, eliminating the influence of self-motion and ensuring that the motion features in the point cloud are entirely derived from real dynamic objects in the environment.

[0014] Further, the degree of change of local point cloud is captured by calculating the distance difference feature m(x, y), which is one of the important features for identifying dynamic objects. If there is motion distortion in the point cloud, the edges of static objects will also show a larger distance difference, causing feature confusion. Motion compensation ensures that the point cloud projected into the same pixel has a true spatial relationship, so that the distance difference feature can accurately reflect the dynamic and static properties of the object, providing reliable input features for the ResNet50 network.

[0015] Furthermore, by accurately timestamping and pose interpolating each scanning point ti, point-level motion compensation accuracy is achieved. This fine processing enables the subsequent dynamic object probability estimation to achieve true pixel-level accuracy. Compared with frame-level rough compensation, point-level compensation avoids boundary ambiguity and detail loss, which is particularly important for detecting small targets moving quickly (such as legs of pedestrians).

[0016] The motion-compensated point cloud accurately reflects the true state of the environment at a certain time, providing a reliable benchmark for subsequent dynamic and static separation. When constructing the occupancy grid map, the dynamic object probability value calculated based on the compensated point cloud can accurately reflect whether the laser beam hits a dynamic object or not, rather than false images caused by motion distortion. This directly affects the accuracy of the map update weights (a and β).

[0017] Furthermore, the adaptive adjustment of the measurement noise covariance matrix in Kalman filtering depends on accurate dynamic object probability values. If the dynamic probability estimation is biased due to motion distortion, the fusion weight will be adjusted incorrectly: in a static environment, the measurement noise may be increased due to distortion, reducing the positioning accuracy; in a dynamic environment, the dynamic degree may be underestimated, making it difficult to effectively suppress interference. Accurate motion compensation ensures the reliability of the dynamic probability value, making the adaptive weight adjustment truly reflect the dynamic characteristics of the environment.

[0018] Further, the dynamic object probability value of each scanning point is obtained by projecting the motion-compensated 3D point cloud data onto a 2D image plane according to a spherical projection method, wherein each pixel position (x, y) corresponds to an elevation angle and a horizontal angle interval.

[0019] For all 3D points projected onto the same pixel position (x, y) wherein, represents the three-dimensional coordinates of the jth 3D point, The measured intensity of the jth 3D point is represented, and the distance r(x, y), the intensity i(x, y), the height h(x, y), and the distance difference m(x, y) are calculated respectively; wherein the distance feature r(x, y) provides basic geometric information; the intensity feature i(x, y) reflects material properties and distinguishes between vehicle metal surfaces and buildings; the height feature h(x, y) identifies ground vehicles and high static objects; and the distance difference feature m(x, y) captures local discontinuities and identifies object boundaries.

[0020] The distance r(x, y), the intensity i(x, y), the height h(x, y), and the distance difference m(x, y) are combined into a 4-channel feature image; the 4-channel feature image is input into a neural network based on the ResNet50 architecture, and through multi-layer convolution feature extraction, a dynamic object score of each pixel position is obtained ; according to the dynamic object score , a dynamic object probability value is obtained through a softmax normalization function .

[0021] In particular, the traditional motion object probability value based on motion consistency, geometric features or statistical distribution is dependent on pre-defined features and thresholds, and the probability value is actually a measure of the degree of deviation from the preset model. In the present application, the motion object probability value is obtained through a neural network, and the probability value output by the softmax reflects the distribution of the data points on the learned feature manifold;

[0022] In a high dynamic environment of a robot (such as a busy street or a densely populated area), the traditional method faces serious challenges: when multiple dynamic objects are mutually occluded, the method based on geometric continuity fails; a slowly moving or pausing pedestrian violates the constant speed motion assumption; different types of dynamic objects (vehicles, pedestrians, bicycles) require different motion models. In the present application, the deep convolutional structure of ResNet50 considers the surrounding environmental information for the probability estimation of each pixel through the receptive field mechanism; and the end-to-end learning enables the model to handle various complex dynamic mode combinations.

[0023] In addition, the traditional rule-based probability is often discrete or piecewise linear, resulting in a sudden change in the Kalman gain. In the present application, the continuous and smooth change of the probability is ensured by the softmax output, so that the state update of the Kalman filter is stable; the probability value of the neural network implicitly contains the joint distribution information of multiple features, rather than the threshold judgment of a single feature. The probability output by the network directly reflects the confidence of the model in the dynamic or static properties of the point, and naturally matches the Bayesian framework of the Kalman filter.

[0024] Furthermore, a static 3D mesh map is obtained by removing dynamic objects, including: projecting motion-compensated 3D point cloud data onto a 3D mesh space of a preset resolution to construct an initial mesh map, wherein the number of laser beam hits α=0 and the number of laser beam passes β=0 for each mesh cell are initialized; and obtaining the maximum detection range of the lidar. The actual measured distance of each laser beam With maximum detection range Compare, The laser beam is defined as the effective returning laser beam. The laser beam is defined as a non-returning laser beam; for each laser beam, based on the probability value of the dynamic object... The ray tracing algorithm is used to determine all the grid cells that the corresponding laser beam passes through from the origin of the lidar to the measurement endpoint;

[0025] For a valid returned laser beam, based on the probability value of the dynamic object... Update the number of laser beam hits α and the number of laser beam passes through each grid cell respectively; for laser beams without return, only update the number of laser beam passes through all grid cells j that the corresponding laser beam passes through: ;in, This indicates the number of times the laser beam passes through the updated mesh cell j; This indicates the number of times the laser beam traversed grid cell j before the update; j represents the index of the grid cell that the laser beam passed through but did not terminate on its path from the start to the end point. ;

[0026] Traverse all laser beams and calculate the reflection probability m of each grid cell based on the updated number of laser beam hits α and the number of laser beam passes β: m = α / (α+β); mark grid cells with a reflection probability m greater than a preset threshold as static objects, and those with a reflection probability m less than a preset threshold as dynamic objects; retain the grid cells marked as static objects and remove the grid cells marked as dynamic objects to obtain a static 3D grid map with dynamic objects removed.

[0027] Furthermore, for the effective returning laser beam, based on the probability value of the dynamic object... Update the number of laser beam hits α and the number of laser beam passes through β of the grid cells respectively, including: ,in, This indicates the number of laser beam hits for the updated grid cell k; This indicates the number of laser beam hits in grid cell k before the update; represents the dynamic object probability value corresponding to the scanning point of the i-th laser beam; k represents the grid cell index where the laser beam terminates. ,in, represents the number of laser beam passes through the updated grid cell k; represents the number of laser beam passes through the updated grid cell k;

[0028] In particular, by ingeniously integrating the dynamic object probability into the Bayesian update framework of the occupancy grid, the probabilistic joint modeling of dynamic and static information is achieved. This soft decision mechanism avoids the information loss of hard threshold segmentation in principle and automatically realizes the separation of dynamic and static through time accumulation.

[0029] Specifically, the traditional occupancy grid map is based on the binary assumption: the laser beam either hits the object (a+1) or passes through the space (b+1). This hard decision ignores the time-varying characteristics of dynamic objects.

[0030] The present scheme introduces dynamic probability Bayesian update: static tendency update: ; dynamic tendency update: The mathematical essence of this soft decision mechanism is to extend the original binomial distribution model to a weighted binomial distribution, where the weight reflects the credibility of the observation. When multiple observations are accumulated, the reflection probability m = a / (a+b) will automatically converge to the true static occupancy state.

[0031] In addition, through the probability weighting mechanism, implicit modeling of the time occupancy pattern of the grid cell is achieved: high static probability accumulation: multiple observations (1-p_dynamic) are accumulated, the value of a grows rapidly, and the reflection probability m tends to 1; high dynamic probability accumulation: accumulation leads to the growth of b value, and the reflection probability m tends to 0; mixed case processing: the grid with partial static and partial dynamic obtains an intermediate probability value; this accumulation effect enables the map to automatically learn the long-term occupancy pattern of each spatial location without explicit time modeling.

[0032] Further, the final positioning result is obtained, including: according to the IMU inertial measurement data and the odometer pose data in the time-aligned sensor data, the dead reckoning is performed through integral operation to obtain a pose prediction value at the current time, the pose prediction value including a position prediction value and an attitude prediction value; the static 3D grid map is taken as a prior map, and the motion-compensated 3D point cloud data at the current time is matched with the static 3D grid map through NDT scanning, and the best registration parameter is solved through iterative optimization to obtain a pose observation value; a state vector of a Kalman filter algorithm is constructed, the state vector including position and attitude information of the robot; the state transition matrix and the process noise covariance matrix are used to predict and update the pose prediction value, to obtain a predicted state estimation value and a predicted error covariance matrix; according to the dynamic object probability value of each scanning point , the measurement noise covariance matrix of the Kalman filter algorithm is adjusted ; adjust the measurement noise covariance matrix of the Kalman filter algorithm , the state estimation value and the error covariance matrix are updated by weighted fusion of the pose prediction value and the pose observation value through the Kalman filter algorithm, and the final positioning result is obtained;

[0033] Further, according to the dynamic object probability value of each scanning point , the measurement noise covariance matrix of the Kalman filter algorithm is adjusted , comprising: .

[0034] In particular, the present application converts the environmental dynamic characteristics into the optimal information fusion weight in the Kalman filter framework, realizing the breakthrough from the fixed noise model to the adaptive noise model.

[0035] Specifically, in the mathematical framework of Kalman filtering, the essence of optimal state estimation is to balance the contributions of prediction value and observation value through Kalman gain K. Among them, the measurement noise covariance matrix R plays a key role - it quantifies the uncertainty of observation and directly determines the allocation of fusion weight. The traditional SLAM system adopts a fixed R value, which implicitly assumes that the measurement noise remains constant in various environments, which may be reasonable in static environments, but becomes a bottleneck for system performance in dynamic environments.

[0036] The influence of dynamic environment on lidar positioning is deep. NDT scan matching relies on the geometric consistency of the current point cloud and the static map to estimate the pose, when there are moving vehicles, pedestrians and other dynamic objects in the scene, the point cloud generated by these objects and the static map generates systematic deviation, causing the matching algorithm to converge to the wrong local optimal solution. This error is not random noise, but systematic error with directionality and correlation, and the fixed measurement noise model cannot accurately describe this complex uncertainty.

[0037] The present application establishes a direct mapping relationship between the dynamic object probability and the measurement uncertainty. Through the dynamic probability output by the deep neural network , the possibility that each scanning point belongs to a dynamic object is accurately quantified, and this probability value essentially reflects the influence degree of the point on the positioning reliability. In static areas, , the measurement noise remains the original value, making full use of the high precision of laser matching; in dynamic areas, , the measurement noise is amplified, automatically reducing the weight of unreliable observations. This continuous probability mapping avoids the instability brought by hard threshold switching, ensuring the smoothness of the system state estimation.

[0038] Further, the distance r(x, y), the intensity i(x, y), the height h(x, y) and the distance difference m(x, y) are calculated respectively, comprising: wherein r(x, y) represents the distance feature value of the pixel position (x, y); represents the jth3D point Euclidean distance to the origin of the lidar; represents the minimum distance value taken from all 3D points projected to the pixel position (x, y); wherein i(x, y) represents the intensity feature value of the pixel position (x, y); represents the measured intensity corresponding to the 3D point P(x, y, j) with the minimum distance; wherein h(x, y) represents the height feature value of the pixel position (x, y); Z(x, y, j) represents the Z coordinate value of the 3D point in the Z coordinate of the lidar coordinate system; wherein m(x, y) represents the distance difference feature value of the pixel position (x, y); represents the neighboring pixel position within the 8-neighborhood of the pixel position (x, y); N represents the number of valid neighboring pixels, N≤8; represents the distance feature value of the neighboring pixel position .

[0039] Another aspect of the present application also provides a deep learning-based adaptive mapping and positioning system, comprising: a data preprocessing module, configured to collect multi-sensor raw data including lidar 3D point cloud data, IMU inertial measurement data and odometer pose data, time-synchronize the multi-sensor raw data to obtain time-aligned sensor data, and perform motion compensation on the lidar 3D point cloud data based on the odometer pose data in the time-aligned sensor data, to obtain motion-compensated 3D point cloud data;

[0040] a dynamic object detection module, configured to project the motion-compensated 3D point cloud data into a multi-channel feature image including distance, intensity, height and distance difference, and perform dynamic object detection using a neural network based on a ResNet50 architecture, to obtain a dynamic object probability value of each scanning point;

[0041] a static map construction module, configured to construct an initial grid map based on the motion-compensated 3D point cloud data, update the number of laser beam hits a and the number of laser beam passes β of each grid cell using the dynamic object probability value, calculate the reflection probability of each grid cell according to the updated number of laser beam hits a and the number of laser beam passes β, and perform dynamic and static object screening on the grid cells in the initial grid map according to the reflection probability, to obtain a static 3D grid map with dynamic objects removed;

[0042] The pose estimation module is configured to obtain a pose prediction value by performing dead reckoning on the time-aligned sensor data, and obtain a pose observation value by performing NDT scan matching on the static 3D grid map;

[0043] The pose fusion module is configured to perform adaptive weighted fusion on the pose prediction value and the pose observation value by using a Kalman filtering algorithm to obtain a final positioning result; wherein the measurement noise covariance matrix of the Kalman filtering algorithm is adjusted according to the dynamic object probability value.

[0044] Compared with the prior art, the application has the following advantages:

[0045] The application introduces pixel-level dynamic object probability estimation based on deep learning, and applies the probability value as a unified weight factor throughout the whole process of map construction and positioning fusion, thereby realizing adaptive optimization of the SLAM system in a dynamic environment.

[0046] Firstly, by projecting the 3D point cloud into a multi-channel feature image containing distance, intensity, height and distance difference, and using a ResNet50 deep neural network for feature learning, a breakthrough is realized from coarse-grained object-level detection to fine-grained pixel-level probability estimation. Each scanning point obtains a continuous dynamic object probability value (between 0 and 1), rather than a traditional binary classification. This probabilistic representation more accurately reflects the dynamic and static attribute uncertainty of the point cloud, and provides more abundant information for subsequent processing.

[0047] Then, the laser beam statistics of the occupancy grid map are differentially weighted and updated based on the dynamic object probability value: for laser beams with low dynamic probability, the contribution of the number of hits is increased , and the representation of static objects in the map is strengthened; for laser beams with high dynamic probability, the contribution of the number of passes is increased , and the influence of dynamic objects is weakened. This probabilistic updating mechanism fundamentally solves the ghosting problem in which the trajectory of a dynamic object is fixed in the map in traditional methods, and constructs a more pure static map.

[0048] Finally, the dynamic object probability value is introduced into the adjustment of the measurement noise covariance matrix of the Kalman filter , and the environmental adaptability of the fusion weight is realized. In a dynamic object dense area, the measurement noise of laser matching is automatically increased, the weight in fusion is reduced, and more reliance is placed on the dead reckoning of the IMU and the odometer; in a static environment, the normal measurement noise is maintained, and the high precision of laser matching is fully utilized. This adaptive mechanism significantly improves the positioning robustness of the system in different dynamic environments. BRIEF DESCRIPTION OF DRAWINGS

[0049] The present application will be further described in the manner of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same numbers represent the same structures, in which:

[0050] Figure 1 is an exemplary flowchart of a deep learning-based adaptive mapping and positioning method according to some embodiments of the present application;

[0051] Figure 2 is a linear interpolation schematic diagram according to some embodiments of the present application;

[0052] Figure 3 is an exemplary flowchart of motion compensation for 3D point cloud data according to some embodiments of the present application. DETAILED DESCRIPTION

[0053] The method and system provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0054] As shown in Figure 1 , 3D laser scanning data is collected by a laser radar, and motion compensation is performed by odometer data to ensure the accuracy of the laser scanning data.

[0055] Real-time dynamic object probability prediction based on a neural network (ResNet50) converts 3D scanning data into a 2D image form, filling multiple modal channels such as range, intensity, and height. A trained neural network is used to process the 2D laser image to predict the probability of each scanning point belonging to a dynamic object. The neural network uses a modified ResNet50 architecture and improves the detection capability of objects of different sizes through a multi-scale convolution module. The network outputs the dynamic object probability of each point through a softmax layer.

[0056] Static 3D grid map construction, based on dynamic object probability, uses a modified grid map construction method proposed by Hähnel et al. to calculate the reflection probability of each grid cell. Specifically, the dynamic probability information is used to update the alpha and beta values of the grid cells, and then the static probability of each cell is calculated. The probability of dynamic objects is directly integrated into the calculation to exclude the influence of dynamic objects and generate a 3D grid map containing only static objects.

[0057] Adaptive Kalman filtering outputs the positioning result. Laser radar data can rely on a stable static map for accurate ndt positioning. The results of odom / imu dead reckoning are subjected to Kalman filtering, the weight of the sensor data is adjusted according to the dynamic object probability, and the positioning accuracy is optimized in real time.

[0058] Specifically, as shown in Figure 2As shown, the system is deployed on an unmanned platform, integrating laser radar, imu, odometer and other sensors. First, time alignment, using ROS Software Synchronization through timestamp linear interpolation (such as or Slerp) to align three kinds of raw data. Through laser radar to collect 3D laser scanning data, through odometer data for motion compensation, to ensure the accuracy of laser scanning data. The compensation process is as follows:

[0059] (1) read laser radar data and odometer data, and perform time synchronization;

[0060] (2) get the start time and end time of the current frame laser radar, get , each time ;

[0061] (3) get the odometer data, stored in the queue in time order, the earliest time in the queue: the earliest odometer data timestamp is less than , the latest timestamp is greater than .

[0062] (4) solve and , the pose of time and ;

[0063] If the odometer data and laser data are completely synchronized, the i, j time corresponds to and : ; .

[0064] At time, there is no corresponding odometer pose, linear interpolation is performed. Set the pose at l, k time, and l < s < k, then ; ; ;

[0065] Use curve approximation to interpolate the pose:

[0066] Use piecewise function to approximate the quadratic curve, when the number of segments is greater than 3, the approximation error can be ignored, and in the time period , a total of k poses are taken. Segment the data (10ms), and get the pose through linear interpolation. Linear interpolation is performed between the k poses, and there are n poses between and , then , then ;

[0067] (5) Coordinate system

[0068] n laser data points are interpolated to get the pose, coordinate transformation ;

[0069] (6) Publish the converted 3D lidar data

[0070] As shown in Figure 3 , this step aims to use a deep learning neural network to process the converted 2D laser image and predict the probability of each scan point belonging to a dynamic object. The neural network is based on the ResNet50 architecture and improves the detection ability of different size objects through multi-scale convolution modules. The dynamic object probability output by the network is used for subsequent dynamic object removal and static map generation.

[0071] First, project the 3D 64-line lidar into a 2D image with a size of 2000x24 pixels. To generate different modalities, first collect all 3D points in the pixel projected to position (x, y): ; where P=(X, Y, Z) is the 3D position and I is the measurement intensity of the 3D point. Using this data, multiple modalities are generated: first, we calculate the minimum range of all 3D points falling into a certain cell: .

[0072] Calculate the average intensity of all points falling into the pixel: ;

[0073] height, calculate the average z value of the pixel, , to remove the absolute distance value, calculate the rangeDiff modality obtained by the standard deviation from the pixel (x, y) to the pixel. Using eight adjacent two-dimensional pixels (x', y') of the cell (x, y) to calculate: .

[0074] For each scan point, the neural network will predict whether it belongs to a dynamic object. In the forward propagation of the neural network, the network will extract image features through multiple convolution modules. Assuming that the final output of the network is the score corresponding to each pixel, indicating the possibility of the point belonging to a dynamic object. For each scan point , the output of the neural network is the score of its belonging to a dynamic object: ; where is the forward propagation function of the neural network, θ is the network parameter, is the input feature of the i-th scan point.

[0075] The softmax layer outputs the dynamic object probability: for each scan point , the network outputs an unnormalized score , which represents the confidence that the point belongs to a dynamic object. This score is converted to a probability by a softmax layer: , where represents the score that the scan point belongs to a dynamic object; represents the score of all classes (static and dynamic) for each pixel; represents the probability that the scan point belongs to a dynamic object.

[0076] The neural network outputs a value between [0, 1] by the softmax function, which represents the probability that the scan point belongs to a dynamic object .

[0077] This probability is further used in the mapping and localization step to remove the influence of dynamic objects.

[0078] Based on the dynamic object probability obtained in step (2), after removing dynamic objects (2D), the prediction results need to be projected back to the 3D scan to construct a static 3D grid map. The expectation maximization (EM) framework is used to determine which laser scan beams are reflected by static objects. The alpha and beta values of each cell in the map are calculated using these scan beams. Alpha represents the number of scan beams ending at the cell, while beta represents the number of scan beams passing through the cell but not ending at the cell. Then, these values are used to calculate the reflection probability of the grid cell according to the following formula: , the continuous dynamic probability is integrated into the calculation of alpha and beta.

[0079] Let p be the dynamic probability calculated by the network, which is applicable to a laser beam that is not a maximum range measurement. For the cell where the laser beam ends, add 1-p to the alpha value. In addition, we add p to the beta value of the cell. If the laser beam is a maximum range measurement, neither the alpha value nor the beta value is updated. Regardless of the maximum range measurement, we add 1 to the beta value of each cell that the laser beam passes through. More formally, for a laser beam with a predicted dynamic probability p, it passes through cell , and ends at cell k, calculate: ;

[0080] The laser beam is not a maximum range measurement, calculate: ; , thus obtaining the static map after removing dynamic objects.

[0081] According to the stable static map obtained, on this map, the Kalman filter is used to weight and fuse the IMU data, lidar data and dynamic object probability.

[0082] Assume a state vector represents the position and pose of the robot at time t. Each time the filter is run, the Kalman filter updates the state estimate based on sensor data, where the pose from the IMU and odometry is the predicted value, and the pose from the LIDAR matching the prior map is the observed value. The update step is as follows:

[0083] The prediction step is: ; ; where is the state transition matrix, is the prediction error covariance matrix, is the process noise.

[0084] When the Kalman filter fuses LIDAR data, the probability of dynamic objects will affect the weighting coefficient, thus adjusting the trustworthiness of sensor data. The system mainly uses the high-precision dynamic map obtained in step (3) to perform NDT scan matching on the LIDAR, and outputs the pose:

[0085] For LIDAR measurements and predicted measurements , the Kalman filter calculates the weighted average: ; where is the measurement matrix, is the measurement noise covariance matrix. The dynamic object probability affects the measurement noise covariance (mainly the LIDAR sensor): ;

[0086] This adjusts the weight of the LIDAR data according to the probability of dynamic objects, so that the Kalman filter makes lower weight adjustments to the measurements of dynamic objects, reducing the interference of dynamic objects on the positioning results.

[0087] The above has described the present application and its embodiments in a schematic manner, which is not restrictive, and the present application can be realized in other specific forms without departing from the spirit or essential characteristics of the present application. The embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it, without departing from the spirit of the present application, similar structural forms and embodiments can be designed without creative design, which shall fall within the protection scope of the present application. In addition, the inclusion of a word does not exclude other elements or steps, and a word before an element does not exclude the inclusion of multiple elements. The words first, second, etc. are used to represent names, and do not represent any specific order.

Claims

1. A deep learning based adaptive mapping and localization method, characterized in that, The method comprises the following steps: Collecting multi-sensor raw data, including laser radar 3D point cloud data, IMU inertial measurement data and odometer pose data; Time synchronization is performed on the multi-sensor raw data to obtain time-aligned sensor data; Based on the odometer pose data in the time-aligned sensor data, the laser radar 3D point cloud data is motion compensated to obtain motion-compensated 3D point cloud data; The motion-compensated 3D point cloud data is projected into a 2D laser image to obtain a multi-channel feature image including distance, intensity, height and distance difference; According to the multi-channel feature image, a neural network based on ResNet50 architecture is used for dynamic object detection to obtain a dynamic object probability value for each scanning point, including: The motion-compensated 3D point cloud data is projected onto a 2D image plane in a spherical projection manner, wherein each pixel position (x, y) corresponds to an elevation angle and a horizontal angle interval; for all 3D points projected to the same pixel position (x, y) wherein, denotes the three-dimensional coordinates of the jth3D point, denotes the measured intensity of the jth3D point, and wherein the distance r(x, y), the intensity i(x, y), the height h(x, y) and the distance difference m(x, y) are calculated, respectively; The distance r(x, y), intensity i(x, y), height h(x, y) and distance difference m(x, y) are combined into a 4-channel feature image; The 4-channel feature image is input into a neural network based on a ResNet50 architecture, and a dynamic object score of each pixel position is obtained through multi-layer convolution feature extraction ; According to the dynamic object score , a dynamic object probability value is obtained by a softmax normalization function ; Based on the motion-compensated 3D point cloud data, an initial grid map is constructed; The dynamic object probability value is used to update the laser beam hit count a and the laser beam pass count β of each grid cell; According to the updated laser beam hit count a and the laser beam pass count β, the reflection probability of each grid cell is calculated; According to the reflection probability, the grid cells in the initial grid map are screened for dynamic and static objects to obtain a static 3D grid map with dynamic objects removed; Track prediction is performed according to the time-aligned sensor data to obtain a pose prediction value; NDT scan matching is performed according to the static 3D grid map to obtain a pose observation value; The Kalman filter algorithm is used to adaptively weight and fuse the pose prediction value and the pose observation value to obtain a final positioning result; wherein the measurement noise covariance matrix of the Kalman filter algorithm is adjusted according to the dynamic object probability value.

2. The adaptive mapping and positioning method based on deep learning according to claim 1, wherein: The motion-compensated 3D point cloud data is obtained, including: Obtain lidar 3D point cloud data and odometer pose data, obtain the start time of a single frame of lidar scanning and the end time , and the timestamp of each scanning point ; The odometer pose data is stored in time sequence to construct a pose time sequence queue; Based on the pose time sequence queue, the LiDAR scanning time period is... Piecewise linear interpolation is performed on the pose of the scan points within the area; Based on the pose obtained from interpolation at each time step, coordinate transformation is performed to convert the pose at each time step... LiDAR scanning points Switch to the start time In the coordinate system, we obtain motion-compensated 3D point cloud data.

3. The adaptive mapping and positioning method based on deep learning according to claim 2, wherein: The static 3D grid map with dynamic objects removed is obtained, including: The motion-compensated 3D point cloud data is projected onto a 3D grid space with a preset resolution to construct an initial grid map, wherein the laser beam hit count a and the laser beam pass count β of each grid cell are initialized as a=0 and β=0; acquiring a maximum detection distance of the laser radar comparing the actual measured distance of each laser beam with the maximum detection distance defining the laser beam with an effective return laser beam, and defining the laser beam with no return laser beam For each laser beam, all grid cells that the corresponding laser beam passes through from the lidar origin to the measurement endpoint are determined using a ray tracing algorithm according to the dynamic object probability value ​ For a valid returned laser beam, based on the probability value of the dynamic object... Update the number of laser beam hits α and the number of laser beam passes through β of the grid cells respectively; For a laser beam without return, only the number of laser beam passes β of all grid cells j through which the corresponding laser beam passes is updated: ; wherein denotes the updated number of laser beam passes of grid cell j; denotes the number of laser beam passes of grid cell j before the update; j denotes the index of the grid cell through which the laser beam passes on the path from the start to the end, but does not end, ; All laser beams are traversed, and the reflection probability m of each grid cell is calculated according to the updated laser beam hit count a and the laser beam pass count β: m=a / (a+β); Grid cells with a reflection probability m greater than a preset threshold are marked as static objects, and vice versa; The grid cells marked as static objects are retained, and the grid cells marked as dynamic objects are removed to obtain a static 3D grid map with dynamic objects removed.

4. The deep learning-based adaptive mapping and localization method according to claim 3, wherein: For an effective return laser beam, the dynamic object probability value is updated according to the following equation: updating the laser beam hit count a and the laser beam pass count b of the laser beam passing through the grid cell, respectively, comprises: wherein, represents the updated number of laser beam hits for grid cell k; represents the updated number of laser beam hits for grid cell k; represents the dynamic object probability value for the i-th laser beam corresponding scanning point; k represents the index of the grid cell where the laser beam terminates. wherein, represents the number of laser beam passes through the updated grid cell k; represents the number of laser beam passes through the updated grid cell k.

5. The deep learning-based adaptive mapping and localization method according to claim 3, wherein: a final localization result is obtained, including: based on the IMU inertial measurement data and the odometry pose data in the time-aligned sensor data, a track prediction is obtained by integral operation, and the track prediction includes a position prediction value and an attitude prediction value; a static 3D grid map is taken as an a priori map, and the motion-compensated 3D point cloud data at the current time is subjected to NDT scan matching with the static 3D grid map, and the best registration parameter is solved by iterative optimization to obtain a pose observation value; a state vector of the Kalman filter algorithm is constructed, and the state vector includes position and attitude information of the robot; the state transition matrix and the process noise covariance matrix are used to predict and update the pose prediction value to obtain a predicted state estimation value and a predicted error covariance matrix; Based on the dynamic object probability value of each scan point Adjust the measurement noise covariance matrix Rt of the Kalman filter algorithm; Utilizing the adjusted measurement noise covariance matrix The state estimation value and the error covariance matrix are updated by the Kalman filtering algorithm to obtain the final positioning result.

6. The deep learning-based adaptive mapping and localization method according to claim 5, wherein: adjusting a measurement noise covariance matrix of a Kalman filter algorithm based on the dynamic object probability value of each scan point comprising: .​ 7. The deep learning-based adaptive mapping and localization method according to claim 2, wherein: the distance r(x, y), the intensity i(x, y), the height h(x, y) and the distance difference m(x, y) are calculated respectively, including: where r(x, y) represents the distance feature value of pixel position (x, y); represents the jth3D point Euclidean distance to the lidar origin; represents the minimum distance value taken over all 3D points that project to pixel position (x, y); where i(x, y) denotes the intensity feature value at pixel position (x, y); denotes the 3D point with the smallest distance the corresponding measured intensity; where h(x, y) represents the height feature value of the pixel position (x, y); represents the 3D point with the minimum distance Z coordinate value in the laser radar coordinate system; where m(x, y) denotes the distance difference feature value of the pixel position (x, y); denotes the neighboring pixel positions within the 8-neighborhood of the pixel position (x, y); N denotes the number of valid neighboring pixels, N < 8; denotes the distance feature value of the neighboring pixel position .

8. A deep learning based adaptive mapping and localization system, characterized in that, including: a data preprocessing module, which collects multi-sensor original data, time-synchronizes the multi-sensor original data to obtain time-aligned sensor data, and motion-compensates the laser radar 3D point cloud data based on the odometry pose data in the time-aligned sensor data to obtain motion-compensated 3D point cloud data; a dynamic object detection module, which projects the motion-compensated 3D point cloud data into a multi-channel feature image, and detects dynamic objects by using a neural network based on a ResNet50 architecture to obtain a dynamic object probability value of each scanning point; a static map construction module, which constructs an initial grid map based on the motion-compensated 3D point cloud data, updates the laser beam hit frequency a and the laser beam passing frequency β of each grid cell by weighting based on the dynamic object probability value, calculates the reflection probability of each grid cell according to the updated laser beam hit frequency a and the laser beam passing frequency β, and screens the grid cells in the initial grid map according to the reflection probability to obtain a static 3D grid map with dynamic objects removed; a pose estimation module, which obtains a pose prediction value by track prediction based on the time-aligned sensor data, and obtains a pose observation value by NDT scan matching based on the static 3D grid map; a pose fusion module, which adaptively weights and fuses the pose prediction value and the pose observation value by using a Kalman filter algorithm to obtain a final localization result; wherein the measurement noise covariance matrix of the Kalman filter algorithm is adjusted according to the dynamic object probability value.

9. A computer-readable storage medium, the storage medium storing computer instructions, when the computer instructions are executed by a processor, implementing the method of any one of claims 1-7.

Images