A multi-source data fusion mapping and positioning method and system, and an autonomous vehicle

Abstract

The application discloses a multi-source data fusion mapping and positioning method and system and an automatic driving vehicle. By acquiring point cloud data and pose information, further obtaining odometer information and positioning, and adding the odometer information into a map to update the map in the front end, according to the key frames in the odometer information and corresponding GPS data, the weights of the odometer factors and GPS factors are constructed and adjusted to update the key frames, and finally, the map is updated in the back end according to the updated key frames. That is, the odometer information is obtained by coupling the point cloud data and the pose information to be added into the map and positioned to ensure continuous and high-frequency mapping and positioning to ensure normal driving of the vehicle. According to the key frame construction and the GPS data, a factor graph is constructed, and the factor graph is optimized to adjust the weights of the odometer factors and the GPS factors, so that the abnormal state and the accumulated error are corrected at a low frequency, and the accuracy and high-precision positioning of multi-source data are realized.

Description

Technical Field

[0001] This application relates to the field of positioning technology for autonomous driving, specifically to a multi-source data fusion mapping and positioning method, system, and autonomous vehicle. Background Technology

[0002] With the continuous development of autonomous driving technology, more and more autonomous vehicles are emerging. However, positioning technology, which is the core technology of autonomous vehicles, remains a challenge. Currently, commonly used positioning methods include: Global Navigation Satellite Systems (GNSS), including GPS and BeiDou; Real-time Differential Technology (RTK), which includes multi-frequency carrier phase differential (centimeter-level) and single-frequency carrier phase differential (sub-meter-level); and Inertial Navigation Technology (INS, IMU).

[0003] Each positioning method has its own advantages and disadvantages. In order to improve positioning accuracy and timeliness, multiple positioning methods can be used simultaneously. However, the refresh rates and data acquisition accuracies of different positioning methods are different, and errors caused by the abnormality of a single sensor may accumulate over a long period of time, resulting in a large final positioning error. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a multi-source data fusion mapping and localization method, system, and autonomous vehicle, thereby resolving the aforementioned technical problems.

[0005] According to one aspect of this application, a multi-source data fusion mapping and positioning method is provided, comprising: acquiring point cloud data; acquiring pose information; obtaining odometer information and performing positioning based on the point cloud data and the pose information; adding the odometer information to a map for front-end map updates; constructing odometer factors based on keyframes in the odometer information; constructing GPS factors based on GPS data corresponding to the odometer information; adjusting the weights of the odometer factors and the GPS factors to update the keyframes; and back-end map updates based on the updated keyframes; wherein the back-end update frequency is lower than the front-end update frequency.

[0006] In one embodiment, obtaining odometer information and performing localization based on the point cloud data and the pose information includes: matching the point cloud data with corresponding historical point cloud data in the map to obtain a laser pose; matching the pose information and the laser pose to obtain a matching residual between the pose information and the laser pose; and when the matching residual is less than a preset residual threshold, performing localization and outputting a driving trajectory; wherein the driving trajectory is determined based on the point cloud data and the pose information.

[0007] In one embodiment, adding the odometer information to the map to update the map from the front end includes: deleting historical point cloud data located outside the visible range of the map when the visible range of the lidar exceeds the boundary of the map; and adding the point cloud data and the pose information to the map to update the map.

[0008] In one embodiment, constructing the odometer factor based on the keyframes in the odometer information includes: selecting keyframes in the odometer information based on the pose information; and constructing the odometer factor based on the keyframes; wherein the odometer factor represents the change matrix of the keyframe relative to the previous frame.

[0009] In one embodiment, constructing the GPS factor based on the GPS data corresponding to the odometer information includes: searching in the GPS buffer for GPS data whose time difference with the odometer information is less than a time threshold; and constructing the GPS factor when the GPS data is a fixed solution.

[0010] In one embodiment, adjusting the weights of the odometer factor and the GPS factor includes: adjusting the weights of the odometer factor and the GPS factor using a reinforcement learning model.

[0011] In one embodiment, the input of the reinforcement learning model includes any one or more of the following three: a depth map obtained by converting the point cloud data corresponding to the current keyframe according to the angle information; the relative transformation of the point cloud data and pose information corresponding to the current keyframe with respect to the previous keyframe; and the relative transformation of the current GPS data corresponding to the current keyframe and the previous GPS data corresponding to the previous keyframe.

[0012] In one embodiment, updating the map based on the updated keyframe includes: adjusting the position of the corresponding point cloud data based on the updated keyframe; and updating the map again based on the adjusted point cloud data.

[0013] According to another aspect of this application, a multi-source data fusion mapping and positioning system is provided, comprising: a point cloud acquisition module for acquiring point cloud data; a pose acquisition module for acquiring pose information; an odometer acquisition module for obtaining odometer information and performing positioning based on the point cloud data and the pose information; a front-end update module for adding the odometer information to a map to update the map from the front end; a first construction module for constructing odometer factors based on keyframes in the odometer information; a second construction module for constructing GPS factors based on GPS data corresponding to the odometer information; a keyframe update module for adjusting the weights of the odometer factors and the GPS factors to update the keyframes; and a back-end update module for updating the map from the back end based on the updated keyframes; wherein the frequency of the back-end update is lower than the frequency of the front-end update.

[0014] According to another aspect of this application, an autonomous vehicle is provided, comprising: a vehicle body; and the aforementioned multi-source data fusion mapping and positioning system, wherein the multi-source data fusion mapping and positioning system is disposed on the vehicle body.

[0015] This application provides a multi-source data fusion mapping and localization method, system, and autonomous vehicle. It acquires point cloud data and pose information, and obtains odometer information for localization based on the point cloud data and pose information. The odometer information is then added to the map for front-end map updates. Next, odometer factors are constructed based on keyframes in the odometer information, and GPS factors are constructed based on the corresponding GPS data. The weights of the odometer and GPS factors are adjusted to update the keyframes. Finally, the map is updated in the back-end based on the updated keyframes. The back-end update frequency is lower than the front-end update frequency. Specifically, odometer information is obtained by coupling point cloud data and pose information to add to the map and perform localization, ensuring continuous and high-frequency mapping and localization to guarantee normal vehicle operation. Furthermore, a factor map is constructed based on keyframes and GPS data, and the factor map is optimized to adjust the weights of the odometer and GPS factors, thereby providing low-frequency correction for abnormal states and accumulated errors, achieving accurate and high-precision localization from multi-source data. Attached Figure Description

[0016] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1This is a flowchart illustrating a multi-source data fusion mapping and positioning method provided in an exemplary embodiment of this application.

[0018] Figure 2 This is a schematic diagram illustrating the principle of a multi-source data fusion mapping and positioning method provided in an exemplary embodiment of this application.

[0019] Figure 3 This is a flowchart illustrating a multi-source data fusion mapping and localization method provided in another exemplary embodiment of this application.

[0020] Figure 4 This is a flowchart illustrating a multi-source data fusion mapping and localization method provided in another exemplary embodiment of this application.

[0021] Figure 5 This is a schematic diagram of the structure of a multi-source data fusion mapping and positioning device provided in an exemplary embodiment of this application.

[0022] Figure 6 This is a schematic diagram of the structure of a multi-source data fusion mapping and positioning system provided in another exemplary embodiment of this application.

[0023] Figure 7 This is a structural diagram of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0024] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0025] As a core technology for autonomous driving, positioning technology needs to achieve real-time and accurate positioning in complex environments with significant interference. Taking smart ports as an example, while the environment within a port is relatively simple, most visible objects are movable or changing, such as quay cranes, yard cranes, containers, trucks, and container ships. These movable or changing objects are often quite large, significantly impacting positioning methods based on historical maps (static maps). Furthermore, due to the abundance of dynamic objects, real-time mapping and localization (SLAM) technology relies on prior assumptions about map points, assuming they are static. However, in practical applications, the current laser scanning point cloud is matched against the historical map. Therefore, if there are too many dynamic points in both the historical map and the current scan frame, it will affect the accuracy of real-time LiDAR point cloud registration, thus impacting mapping and positioning performance. Moreover, the port environment is highly susceptible to interference, with single sensors prone to failure and RTK signals unstable. For example, obstructions under quay cranes can lead to the loss of fixed solutions, potentially resulting in RTK positioning drift. Weather interference affects sensors. For example, in rainy weather, the noise of the lidar increases, and the presence of water on the ground prevents the lidar points from reflecting, resulting in the loss of ground point clouds and the loss of ground plane constraints. This can easily lead to SLAM degradation in monotonous "long corridor" environments.

[0026] The above-mentioned problems may lead to a decrease in the accuracy of multi-sensor data fusion localization due to the failure of a single sensor. To solve this technical problem, this application provides a multi-source data fusion mapping and localization method and system. It performs high-frequency front-end mapping and localization (i.e., real-time mapping and localization) using point cloud data and pose information acquired at the front end to meet the high-frequency localization requirements of autonomous driving. Simultaneously, it optimizes the factor map of multiple sensors at the back end to adjust the weight parameters of each sensor, reducing the impact of errors caused by interference from abnormal sensors on the overall localization result. Furthermore, based on the back end's correction of abnormal states and long-term accumulated errors, while ensuring real-time mapping and localization, it utilizes the back end to adjust the weight parameters of the sensors to correct accumulated errors, thereby improving localization accuracy.

[0027] The implementation of the multi-source data fusion mapping and positioning method and system provided in the embodiments of this application is described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a schematic flowchart illustrating a multi-source data fusion mapping and localization method provided in an exemplary embodiment of this application. Figure 1 As shown, the multi-source data fusion mapping and localization method includes the following steps:

[0029] Step 110: Obtain point cloud data.

[0030] Specifically, point cloud data is collected in real time using lidar. For example, the radar points within a time frame of 10 to 100 milliseconds can be accumulated as a scan input (scan frame), corresponding to a frequency of 100 Hz to 10 Hz.

[0031] Step 120: Obtain pose information.

[0032] Specifically, an inertial measurement unit (IMU) is used to acquire pose information in real time. Based on the pose information acquired by the IMU, the pose changes between frames are calculated. At the same time, based on the pose information from the IMU and the time of each laser point, the distance information of the laser point is converted into coordinate information.

[0033] Step 130: Obtain odometry information and perform positioning based on point cloud data and pose information.

[0034] Specifically, such as Figure 2 As shown, points from a single scan can be registered in a large local map using a tightly-coupled iterated Kalman filter (ESEKF), while simultaneously estimating odometry information. When an update command is received from the backend, the state values ​​in the ESEKF are updated, such as position, attitude, and interference values ​​for each sensor.

[0035] Step 140: Add the odometer information to the map and update the map from the front end.

[0036] Specifically, global map points are stored in the ikd-tree (incremental kd tree) data format. After the odometry outputs point cloud data and pose information, the odometry outputs point cloud data and pose information are integrated into the map by inserting the ikd-tree according to the odometry refresh rate. That is, the collected point cloud data and pose information are added to the map incrementally.

[0037] Step 150: Construct the odometer factor based on the keyframes in the odometer information.

[0038] In one embodiment, step 150 can be implemented as follows: based on pose information, keyframes are selected from the odometry information, and odometry factors are constructed based on the keyframes; wherein, the odometry factor represents the change matrix of the keyframe relative to the previous frame. Specifically, keyframes are selected based on pose information and RTK deviation values, and a keyframe map is constructed. The relative position of the current keyframe to the previous keyframe is calculated to construct odometry factors, which are then added to the factor map optimization. The selection of keyframes can be based on manually set rules, such as moving a certain distance, turning a certain angle, or comparing the position with RTK or other global positioning inputs. If the difference is too large, it can be considered a keyframe.

[0039] Optionally, such as Figure 2 As shown, loop closure detection features can also be constructed with keyframes, and it can be determined whether the current keyframe and the historical keyframe are close in position. If the position distance between the current keyframe and the historical keyframe is less than the first preset distance, loop closure detection feature matching is triggered. If the matching degree is high (the position distance between the current keyframe and the historical keyframe is less than the second preset distance, where the second preset distance is less than the first preset distance), the point cloud is precisely matched, and the relative position with the historical keyframe is output as a loop closure detection factor and added to the factor map optimization.

[0040] Step 160: Construct GPS factors based on the GPS data corresponding to the odometer information.

[0041] In one embodiment, step 160 can be implemented as follows: Searching for GPS data in the GPS buffer whose time difference with the odometer information is less than a time threshold; when the GPS data is a fixed solution, constructing a GPS factor. Specifically, searching for data in the GPS buffer whose time is close to the time of the odometer information; if this data is a fixed solution, constructing a GPS factor and adding it to the factor graph optimization.

[0042] Step 170: Adjust the weights of the odometer factor and the GPS factor to update the keyframe.

[0043] Specifically, the weights of the factors corresponding to each sensor in the factor graph are obtained through factor graph optimization. These weights can be the weights of the interference factors corresponding to each sensor. After obtaining the weights, the point cloud data corresponding to each key frame is changed according to the pose of the key frame to update the key frame.

[0044] Step 180: The backend updates the map based on the updated keyframes.

[0045] The backend updates less frequently than the frontend. In one embodiment, step 180 can be implemented by adjusting the position of the corresponding point cloud data based on the updated keyframes, and then updating the map again based on the adjusted point cloud data. Specifically, as shown... Figure 2As shown, after updating the keyframe, the backend sends the updated keyframe to the frontend for map reconstruction (i.e., map update) to correct abnormal states and long-term accumulated errors. It should be understood that the frontend and backend in this application can be two processes or modules integrated into a single processor, or they can be two independent processors communicating with each other. The frontend in this application uses a tightly coupled ESKF-based lidar and IMU odometry for real-time mapping and positioning. The backend is based on multi-sensor fusion weight optimization using factor maps. That is, the frontend achieves real-time mapping and positioning, while the backend performs weight optimization and adjustment on the multi-source data acquired from multiple sensors to correct abnormal states and accumulated errors in the keyframe, thereby improving positioning accuracy.

[0046] This application provides a multi-source data fusion mapping and positioning method. It acquires point cloud data and pose information, then obtains odometer information and performs positioning based on the point cloud data and pose information. The odometer information is added to the map for front-end map updates. Next, odometer factors are constructed based on keyframes in the odometer information, and GPS factors are constructed based on the corresponding GPS data. The weights of the odometer and GPS factors are adjusted to update the keyframes. Finally, the map is updated on the back-end based on the updated keyframes. The back-end update frequency is lower than the front-end update frequency. That is, odometer information is obtained by coupling point cloud data and pose information to add to the map and perform positioning, ensuring continuous and high-frequency mapping and positioning to guarantee normal vehicle operation. Furthermore, a factor map is constructed based on keyframes and GPS data, and the factor map is optimized to adjust the weights of the odometer and GPS factors, thereby performing low-frequency corrections for abnormal states and accumulated errors, achieving accurate and high-precision positioning based on multi-source data.

[0047] Figure 3 This is a flowchart illustrating a multi-source data fusion mapping and localization method provided in another exemplary embodiment of this application. Figure 3 As shown, step 130 above may include:

[0048] Step 131: Match the point cloud data with the corresponding historical point cloud data in the map to obtain the laser pose.

[0049] The point cloud data is distorted, and the distorted point cloud data is matched with the historical point cloud map to obtain the laser pose.

[0050] Step 132: Match the pose information and the laser pose to obtain the matching residual between the pose information and the laser pose.

[0051] Calculate the matching residual between the pose information and the laser pose, that is, the difference between the calculated pose information and the laser pose.

[0052] Step 133: When the matching residual is less than the preset residual threshold, perform positioning and output the driving trajectory.

[0053] The driving trajectory is determined based on point cloud data and pose information. If the matching residual is less than a preset residual threshold, it indicates that the matching residual is small, meaning that the point cloud data collected by the LiDAR and the pose information collected by the IMU have converged. At this point, the trajectory is output and the system waits for the next input of point cloud data. If the matching residual is greater than or equal to the residual threshold, it indicates that the point cloud data collected by the LiDAR and the pose information collected by the IMU have not converged. In this case, the LiDAR pose can be re-estimated, and the above process can be repeated until the point cloud data collected by the LiDAR and the pose information collected by the IMU converge.

[0054] Figure 4 This is a flowchart illustrating a multi-source data fusion mapping and localization method provided in another exemplary embodiment of this application. Figure 4 As shown, step 140 above may include:

[0055] Step 141: When the visible range of the LiDAR exceeds the boundary of the map, delete the historical point cloud data located outside the visible range in the map.

[0056] Step 142: Add point cloud data and pose information to the map to update the map.

[0057] Specifically, a cube is constructed based on the LiDAR's field of view. As the vehicle moves, the cube moves along with it while maintaining its size. If the cube exceeds the current map boundary, it means that the map needs to be updated to include the area containing the cube. In this case, historical points that are far away (such as points outside the cube) can be deleted from the map to reduce map data and improve map accuracy.

[0058] In one embodiment, step 170 can be implemented by using a reinforcement learning model to adjust the weights of the odometer factor and the GPS factor.

[0059] Specifically, the input to the reinforcement learning model includes any one or more of the following three combinations: the depth map obtained by transforming the point cloud data corresponding to the current keyframe based on the angle information; the relative transformation of the point cloud data and pose information corresponding to the current keyframe relative to the previous keyframe; and the relative transformation of the current GPS data corresponding to the current keyframe and the previous GPS data corresponding to the previous keyframe.

[0060] A reinforcement learning model is used to solve the factor graph optimization problem to obtain the weights of multiple sensors, such as the odometry factor and the GPS factor. Specifically, the input of the reinforcement learning model can be obtained based on the observations of the current environmental state. That is, the weights of each sensor are adaptively adjusted according to the current environmental state. For example, if the relative change of the point cloud data corresponding to the current keyframe is large compared to the previous keyframe, it may be that the current point cloud data is abnormal. In this case, the weights of the LiDAR can be adjusted appropriately to reduce the impact of the LiDAR on the positioning results.

[0061] Furthermore, this reinforcement learning model can use Multi-Layer Perceptorn (MLPs) as the network backbone and construct a two-layer MLP backbone structure. The training method for this reinforcement learning model can be as follows: the training set is split into training samples of 10 keyframes per group, and the reward for each iteration is the relative positional error between the localization result and the actual localization data. The Twin Delayed Deep Deterministic policy gradient algorithm (TD3) is used as the maximized long-term reward to ensure continuous localization accuracy.

[0062] Figure 5 This is a schematic diagram of the structure of a multi-source data fusion mapping and positioning system provided in an exemplary embodiment of this application. For example... Figure 5 As shown, the multi-source data fusion mapping and positioning system 50 includes: a point cloud acquisition module 51 for acquiring point cloud data; a pose acquisition module 52 for acquiring pose information; an odometer acquisition module 53 for obtaining odometer information and performing positioning based on the point cloud data and pose information; a front-end update module 54 for adding odometer information to the map to update the map from the front end; a first construction module 55 for constructing odometer factors based on keyframes in the odometer information; a second construction module 56 for constructing GPS factors based on GPS data corresponding to the odometer information; a keyframe update module 57 for adjusting the weights of the odometer factors and GPS factors to update the keyframes; and a back-end update module 58 for updating the map from the back end based on the updated keyframes; wherein the frequency of back-end updates is lower than the frequency of front-end updates.

[0063] This application provides a multi-source data fusion mapping and positioning system. A point cloud acquisition module 51 acquires point cloud data, a pose acquisition module 52 acquires pose information, and an odometer acquisition module 53 obtains odometer information and performs positioning based on the point cloud data and pose information. Furthermore, a front-end update module 54 adds the odometer information to the map for front-end map updates. Then, a first construction module 55 constructs odometer factors based on keyframes in the odometer information, a second construction module 56 constructs GPS factors based on the GPS data corresponding to the odometer information, and a keyframe update module 57 adjusts the odometer factors and GPS factors. The weights are adjusted to update keyframes. Finally, the backend update module 58 updates the map based on the updated keyframes. The backend update frequency is lower than the frontend update frequency. That is, the odometer information is obtained by coupling point cloud data and pose information to add to the map and perform positioning, so as to ensure continuous and high-frequency mapping and positioning to ensure normal vehicle operation. Furthermore, a factor map is constructed based on keyframes and GPS data, and the factor map is optimized to adjust the weights of the odometer factor and GPS factor, thereby performing low-frequency corrections for abnormal states and cumulative errors to achieve accurate and high-precision positioning of multi-source data.

[0064] In one embodiment, the first construction module 55 may be further configured to: select key frames from the odometer information based on the pose information, and construct odometer factors based on the key frames; wherein the odometer factors represent the change matrix of the key frame relative to the previous frame.

[0065] In one embodiment, the second construction module 56 can be further configured to: search for GPS data in the GPS buffer whose time difference with the odometer information is less than a time threshold, and construct a GPS factor when the GPS data is a fixed solution.

[0066] In one embodiment, the backend update module 58 can be further configured to: adjust the position of the corresponding point cloud data according to the updated keyframe, and update the map again according to the adjusted point cloud data.

[0067] Figure 6 This is a schematic diagram of the structure of a multi-source data fusion mapping and positioning system provided in another exemplary embodiment of this application. For example... Figure 6 As shown, the odometer acquisition module 53 may include: a point cloud matching unit 531, used to match point cloud data with corresponding historical point cloud data in the map to obtain laser pose; a residual calculation unit 532, used to match pose information and laser pose to obtain the matching residual between pose information and laser pose; and a trajectory output unit 533, used to perform positioning and output driving trajectory when the matching residual is less than a preset residual threshold; wherein, the driving trajectory is determined based on point cloud data and pose information.

[0068] In one embodiment, the front-end update module 54 can be further configured to: when the visible range of the lidar exceeds the boundary of the map, delete the historical point cloud data located outside the visible range in the map, and add the point cloud data and pose information to the map to update the map.

[0069] In one embodiment, the keyframe update module 57 may be further configured to: adjust the weights of the odometry factor and the GPS factor using a reinforcement learning model. In one embodiment, the input to the reinforcement learning model includes any one or more of the following three: a depth map obtained by converting the point cloud data corresponding to the current keyframe according to the angle information; the relative transformation of the point cloud data and pose information corresponding to the current keyframe relative to the previous keyframe; and the relative transformation of the current GPS data corresponding to the current keyframe and the previous GPS data corresponding to the previous keyframe.

[0070] This application provides an autonomous vehicle, including: a vehicle body; and the aforementioned multi-source data fusion mapping and positioning system, which is disposed on the vehicle body.

[0071] This application provides an autonomous vehicle that acquires point cloud data and pose information, obtains odometer information based on the point cloud data and pose information, and performs localization. The odometer information is then added to a map for front-end map updates. Next, odometer factors are constructed based on keyframes in the odometer information, and GPS factors are constructed based on the corresponding GPS data. The weights of the odometer and GPS factors are adjusted to update the keyframes. Finally, the map is updated back-end based on the updated keyframes. The back-end update frequency is lower than the front-end update frequency. Specifically, odometer information is obtained by coupling point cloud data and pose information to add to the map and perform localization, ensuring continuous and high-frequency mapping and localization to guarantee normal vehicle operation. Furthermore, a factor map is constructed based on keyframes and GPS data, and the factor map is optimized to adjust the weights of the odometer and GPS factors, thereby performing low-frequency corrections for abnormal states and accumulated errors to achieve accurate and high-precision localization from multi-source data.

[0072] Below, for reference Figure 7 This application describes an electronic device according to embodiments thereof. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0073] Figure 7 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0074] like Figure 7As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0075] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0076] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this application described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0077] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0078] When the electronic device is a standalone device, the input device 13 can be a communication network connector for receiving the collected input signals from the first device and the second device.

[0079] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0080] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0081] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0082] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0083] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0084] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for mapping and localization based on multi-source data fusion, characterized in that, include: Acquire point cloud data; Obtain pose information; Based on the point cloud data and the pose information, the odometer information is obtained and the location is performed; The odometer information is added to the map, and the map is updated from the front end. Based on the keyframes in the odometer information, an odometer factor is constructed; Based on the GPS data corresponding to the odometer information, a GPS factor is constructed; Adjust the weights of the odometer factor and the GPS factor to update the keyframe; as well as Based on the updated keyframes, the backend updates the map; The frequency of backend updates is lower than the frequency of frontend updates; The adjustment of the weights of the odometer factor and the GPS factor includes: The weights of the odometry factor and the GPS factor are adjusted using a reinforcement learning model; The input to the reinforcement learning model includes any one or more of the following three combinations: The depth map is obtained by converting the point cloud data corresponding to the current keyframe based on the angle information. The relative transformation of the point cloud data and pose information corresponding to the current keyframe relative to the previous keyframe; and The relative change between the current GPS data corresponding to the current keyframe and the previous GPS data corresponding to the previous keyframe.

2. The multi-source data fusion mapping and positioning method according to claim 1, characterized in that, The step of obtaining odometry information and performing positioning based on the point cloud data and the pose information includes: The point cloud data is matched with the corresponding historical point cloud data in the map to obtain the laser pose; Match the pose information and the laser pose to obtain the matching residual between the pose information and the laser pose; and When the matching residual is less than a preset residual threshold, positioning is performed and the driving trajectory is output; wherein, the driving trajectory is determined based on the point cloud data and the pose information.

3. The multi-source data fusion mapping and positioning method according to claim 1, characterized in that, The step of adding the odometer information to the map and updating the map from the front end includes: When the visible range of the lidar exceeds the boundary of the map, delete historical point cloud data located outside the visible range from the map; and The point cloud data and the pose information are added to the map to update the map.

4. The multi-source data fusion mapping and positioning method according to claim 1, characterized in that, The step of constructing the odometer factor based on the keyframes in the odometer information includes: Based on the pose information, keyframes are selected from the odometer information; and Based on the keyframe, an odometer factor is constructed; wherein the odometer factor represents the change matrix of the keyframe relative to the previous frame.

5. The multi-source data fusion mapping and positioning method according to claim 1, characterized in that, The step of constructing GPS factors based on the GPS data corresponding to the odometer information includes: Search the GPS cache for GPS data whose time difference with the odometer information is less than a time threshold; When the GPS data is a fixed solution, the GPS factor is constructed.

6. The multi-source data fusion mapping and positioning method according to claim 1, characterized in that, The step of updating the map based on the updated keyframes includes: Based on the updated keyframes, adjust the positions of the corresponding point cloud data; and The map is updated again based on the point cloud data after the position is adjusted.

7. A multi-source data fusion mapping and positioning system, characterized in that, include: The point cloud acquisition module is used to acquire point cloud data; The pose acquisition module is used to acquire pose information; The odometer acquisition module is used to obtain odometer information and perform positioning based on the point cloud data and the pose information; The front-end update module is used to add the odometer information to the map so that the map is updated from the front end; The first construction module is used to construct odometer factors based on key frames in the odometer information; The second construction module is used to construct GPS factors based on the GPS data corresponding to the odometer information; A keyframe update module is used to adjust the weights of the odometer factor and the GPS factor in order to update the keyframe; The adjustment of the weights of the odometry factor and the GPS factor includes: adjusting the weights of the odometry factor and the GPS factor using a reinforcement learning model; the input of the reinforcement learning model includes any one or more of the following three: a depth map obtained by transforming the point cloud data corresponding to the current keyframe according to angle information; the relative transformation of the point cloud data and pose information corresponding to the current keyframe relative to the previous keyframe; and the relative transformation of the current GPS data corresponding to the current keyframe and the previous GPS data corresponding to the previous keyframe; and The backend update module is used to update the map based on the updated keyframes. The frequency of backend updates is lower than the frequency of frontend updates.

8. An autonomous vehicle, characterized in that, include: Vehicle body; as well as The multi-source data fusion mapping and positioning system as described in claim 7, wherein the multi-source data fusion mapping and positioning system is disposed on the vehicle body.

Images