A map construction method and device, electronic equipment and storage medium

Abstract

The application discloses a map construction method and device, electronic equipment and a storage medium, and relates to the technical field of data fusion. In the application, radar point cloud data and image acquisition data collected from a to-be-measured area are acquired, and the radar point cloud data and the image acquisition data are analyzed respectively to obtain a first measurement pose and a second measurement pose of a map construction device in a standard coordinate reference system, so that a target fusion pose of the map construction device is obtained based on the first measurement pose and the second measurement pose, and then a map of the to-be-measured area is constructed based on the target fusion pose, the radar point cloud data and the image acquisition data. In this way, the technical defect that the accuracy of map construction is low due to the fact that the poses of the map construction devices determined by a laser SLAM constructed map and a visual SLAM constructed map are different is avoided, so that tight coupling of multiple sensors is achieved, and the accuracy of map construction is improved.

Description

Technical Field

[0001] This application relates to the field of data fusion technology, and in particular to a map construction method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development of information technology and data processing technology, there are more and more methods for building maps. Among them, Simultaneous Localization and Mapping (SLAM) is the most commonly used method. It usually refers to the localization of a robot's own position and pose (i.e., pose) and the construction of a map by collecting and calculating data from various sensors on a robot or other carrier. It is widely used in fields such as intelligent driving, service robots, and drones.

[0003] Currently, the two main SLAM technologies are laser SLAM and visual SLAM. However, visual SLAM is easily affected by factors such as ambient light and camera movement. Furthermore, laser SLAM has low vertical resolution and can only acquire sparse point cloud information, resulting in limited environmental features.

[0004] Therefore, in order to improve the accuracy of map construction, laser SLAM and visual SLAM technologies can be combined to build maps, that is, to merge the map built using laser SLAM and the map built using visual SLAM.

[0005] However, the map building method described above does not take into account that the maps built by laser SLAM and vision SLAM are built based on the poses of the robot or other carriers, which reduces the accuracy of map building when the two poses are different.

[0006] Therefore, the accuracy of map construction using the above method is relatively low. Summary of the Invention

[0007] This application provides a map building method, apparatus, electronic device, and storage medium to improve the accuracy of map building.

[0008] In a first aspect, embodiments of this application provide a map building method applied to a map building device, the method comprising:

[0009] From the area to be tested, acquire the collected radar point cloud data and image acquisition data;

[0010] The radar point cloud data is analyzed to obtain the first measurement pose of the map building device in the standard coordinate reference system;

[0011] The image acquisition data is analyzed to obtain the second measurement pose of the map building device in the standard coordinate reference system;

[0012] Based on the first and second measurement poses, the target fusion pose of the map building device is obtained;

[0013] A map of the area to be measured is constructed based on target fusion pose, radar point cloud data, and image acquisition data.

[0014] Secondly, embodiments of this application also provide a map building apparatus, applied to a map building device, the apparatus comprising:

[0015] The acquisition module is used to acquire radar point cloud data and image acquisition data from the area to be tested;

[0016] The parsing module is used to parse radar point cloud data to obtain the first measurement pose of the map building device in the standard coordinate reference system, and to parse image acquisition data to obtain the second measurement pose of the map building device in the standard coordinate reference system.

[0017] The fusion module is used to obtain the target fusion pose of the map building device based on the first and second measurement poses.

[0018] The module is used to build a map of the area to be tested based on target fusion pose, radar point cloud data, and image acquisition data.

[0019] In an optional embodiment, when obtaining the target fusion pose of the map building device based on the first and second measurement poses, the fusion module is specifically used for:

[0020] Using capacitive Kalman filtering, the first and second measurement poses are predicted respectively to obtain the corresponding first target state information set and second target state information set;

[0021] Based on the first target state information set and the second target state information set, the target fusion pose of the map building device is obtained.

[0022] In one optional embodiment, the first target state information set includes: a target state information matrix, a target state information vector, a first state information increment, and a first association information matrix; the second target state information set includes: a target state information matrix, a target state information vector, a second state information increment, and a second association information matrix.

[0023] When obtaining the target fusion pose of the map building device based on the first target state information set and the second target state information set, the fusion module is specifically used for:

[0024] Based on the first state information increment and the second state information increment, the corresponding fused state information increment is obtained, and based on the first association information matrix and the second association information matrix, the corresponding fused association information matrix is ​​obtained.

[0025] Based on the target state information matrix, target state information vector, fusion state information increment, and fusion association information matrix, the target fusion pose of the map building device is obtained.

[0026] In one embodiment, during the process of acquiring collected radar point cloud data and image acquisition data from the area to be tested, the acquisition module is further configured to:

[0027] The third and fourth measurement poses of the map building device, respectively, are acquired from the area to be measured by the inertial measurement unit and the wheeled odometer.

[0028] In an optional embodiment, when obtaining the target fusion pose of the map building device based on the first target state information set and the second target state information set, the fusion module is further configured to:

[0029] Based on the preset Kalman filter algorithm, the third and fourth measurement poses are fused to obtain the corresponding initial fused pose.

[0030] The capacitive Kalman filter algorithm is used to predict the initial fusion pose and obtain the corresponding initial fusion state information set.

[0031] Based on the initial fusion state information set, the first target state information set, and the second target state information set, the target fusion pose of the map building device is obtained.

[0032] In one optional embodiment, when constructing a map of the area to be measured based on target fusion pose, radar point cloud data, and image acquisition data, the construction module is specifically used for:

[0033] Obtain the corresponding map requirement type from the map building request;

[0034] Based on the map construction method set according to the corresponding map requirement type, the radar point cloud data and image acquisition data are adjusted to obtain the adjusted radar point cloud data and image acquisition data.

[0035] A map of the area to be measured is constructed based on the target fusion pose and adjusted radar point cloud data and image acquisition data.

[0036] Thirdly, embodiments of this application also propose an electronic device including a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of the map construction method described in the first aspect.

[0037] Fourthly, embodiments of this application also propose a computer-readable storage medium comprising program code, which, when executed on an electronic device, causes the electronic device to perform the steps of the map construction method described in the first aspect.

[0038] Fifthly, embodiments of this application also provide a computer program product, which, when invoked by a computer, causes the computer to execute the map construction method steps as described in the first aspect.

[0039] The beneficial effects of this application are as follows:

[0040] In the map building method provided in this application embodiment, the target fusion pose of the map building device is obtained according to the first measurement pose and the second measurement pose. Then, based on the target fusion pose, radar point cloud data and image acquisition data, a map of the area to be measured is built. This avoids the problem in related technologies that do not take into account that the maps built by each sensor are built on the pose of their respective map building devices. Therefore, it realizes tight-coupled map building of multiple sensors and improves the accuracy of map building.

[0041] Furthermore, other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described herein are used to provide a further understanding of this application, constitute a part of this application, and do not constitute an improper limitation of this application. In the accompanying drawings:

[0043] Figure 1 This is an optional schematic diagram of the system architecture applicable to the embodiments of this application;

[0044] Figure 2 A schematic diagram of a map building device and its various sensors provided in this application embodiment;

[0045] Figure 3 A schematic diagram illustrating the implementation process of a map construction method provided in this application embodiment;

[0046] Figure 4 A method based on the embodiments of this application is provided. Figure 3 Specific application scenario diagram;

[0047] Figure 5 A logical schematic diagram illustrating how to obtain the target fusion pose of a map building device, as provided in an embodiment of this application;

[0048] Figure 6 This application provides a logical diagram illustrating a map construction method.

[0049] Figure 7 This is a schematic diagram of the structure of a map building device provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0052] It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0053] Furthermore, the data collection, dissemination, and use in the technical solution of this application all comply with the requirements of relevant national laws and regulations.

[0054] The following explanations of some technical terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0055] (1) Inertial Measurement Unit (IMU): is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object.

[0056] It should be noted that an inertial measurement unit (IMU) contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the acceleration signals of the object in the three independent axes of the carrier coordinate system, while the gyroscopes detect the angular velocity signals of the carrier relative to the navigation coordinate system. The IMU measures the angular velocity and acceleration of the object in three-dimensional space and uses this to calculate the object's attitude.

[0057] (2) Tight coupling: Data is put together for joint optimization, that is, the relationship between modules or systems is relatively close and there are mutual calls.

[0058] (3) Loose coupling: Estimate the parameters separately and then fuse the multiple parameters, for example, the simplest weighted average, or using a Kalman filter.

[0059] (4) Real-Time Appearance-Based Mapping (RTAB-Map): This is an open-source library that implements loop closure detection through memory management methods. By limiting the size of the map, loop closure detection is always processed within a fixed time limit, thereby meeting the online mapping requirements of long-term and large-scale environments.

[0060] Furthermore, based on the above explanations of terms and related terminology, the design concept of the embodiments of this application will be briefly introduced below:

[0061] Currently, as the methods for building maps have gradually increased, SLAM is the most commonly used method. It usually refers to the localization of a robot's own position and pose (i.e., pose) and the construction of a map by collecting and calculating data from various sensors on a robot or other carrier. It is widely used in fields such as intelligent driving and map building in unknown environments.

[0062] However, current SLAM methods using a single sensor all have certain drawbacks. Pure vision SLAM is highly susceptible to the effects of ambient light and camera movement, while pure laser SLAM has low vertical resolution and can only acquire sparse point cloud information, resulting in limited environmental features. Furthermore, the positioning effect relying solely on inertial measurement units and trajectory extrapolation also suffers from significant cumulative errors. In addition, current multi-sensor fusion methods focus on the fusion of vision and IMU, LiDAR and IMU, and vision and LiDAR.

[0063] Among them, the fusion of LiDAR and IMU can overcome the problems of low vertical resolution, slow update rate and distortion caused by motion in LiDAR SLAM, but it is not suitable for 3D environment map building; the fusion of LiDAR and vision can improve the positioning accuracy and robustness of SLAM, but the positioning and map building accuracy is very poor when the vehicle is moving at high speed and turning; the fusion of vision and IMU can effectively improve the positioning and map building accuracy of the vehicle during movement, but the cumulative error is large.

[0064] It is evident that multi-sensor fusion can further improve the accuracy of localization and map building. However, most existing multi-sensor fusion methods adopt a loosely coupled approach, that is, after using a certain sensor for localization and map building, the built maps are fused together, such as RTAB-Map. This approach does not fuse the pose information of each sensor during the process, and still has a large error.

[0065] It is obvious that when there are significant differences in the pose information of each sensor, directly fusing the maps built by each sensor will result in a low accuracy of the final fused map.

[0066] In view of this, in order to further improve the positioning and mapping accuracy of SLAM in complex environments, this application proposes a multi-sensor map building method, such as a map building method using LiDAR and image acquisition devices. Specifically, for any sampling time of the LiDAR and image acquisition devices, the method includes: acquiring radar point cloud data and image acquisition data collected from the area to be measured by the LiDAR and image acquisition devices respectively at the current sampling time; then, parsing the radar point cloud data to obtain the first measurement pose of the map building device in the standard coordinate reference system, and parsing the image acquisition data to obtain the second measurement pose of the map building device in the standard coordinate reference system; further, based on the first and second measurement poses, obtaining the target fusion pose of the map building device; finally, based on the target fusion pose, radar point cloud data, and image acquisition data, constructing a map of the area to be measured, thereby improving the accuracy of map building.

[0067] In particular, the preferred embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0068] See Figure 1The diagram illustrates a system architecture applicable to an embodiment of this application. This system architecture includes a target terminal (101a, 101b) and a server 102. The target terminal (101a, 101b) and the server 102 can interact via a communication network. The communication network can employ wireless communication or wired communication methods.

[0069] For example, the target terminal (101a, 101b) can access the network and communicate with the server 102 through cellular mobile communication technology, wherein the cellular mobile communication technology includes, for example, 5th generation mobile network (5G) technology.

[0070] Optionally, the target terminal (101a, 101b) can access the network and communicate with the server 102 via short-range wireless communication, wherein the short-range wireless communication method includes, for example, Wireless Fidelity (Wi-Fi) technology.

[0071] This application embodiment does not impose any limitation on the number of communication devices involved in the above system architecture, such as Figure 1 As shown, only the target terminal (101a, 101b) and server 102 are described as examples. The following is a brief introduction to each of the above devices and their respective functions.

[0072] The target terminal (101a, 101b) is a device that can provide voice and / or data connectivity to a user, and can be a device that supports wired and / or wireless connection methods.

[0073] For example, the target terminals (101a, 101b) include, but are not limited to: mobile phones, tablets, laptops, handheld computers, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in autonomous driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.

[0074] Furthermore, the target terminals (101a, 101b) may have related client software installed. This client software may be an application (APP), browser, short video software, or a webpage, mini-program, etc. In this embodiment, the target terminals (101a, 101b) can be used to send data information collected by various sensors (such as lidar, image acquisition devices, inertial measurement units, and wheeled odometers) to the server 102.

[0075] In one alternative embodiment, see [reference] Figure 2 As shown, the target terminal can be a map building device 20, which can carry various sensors, including: lidar 201, image acquisition device 202, inertial measurement unit 203 and wheeled odometer 204, to collect environmental information of the area to be measured 21 for subsequent map building.

[0076] Among them, the lidar 201 can collect sparse point cloud information (i.e., radar point cloud data) in the plane corresponding to the area to be measured 21, the image acquisition device 202 can be used to collect the color information and depth information of the environment in the area to be measured 21 (i.e., image acquisition data), the inertial measurement unit 203 can be used to collect the three-axis acceleration information and angular velocity of the map building device 20, and the wheel odometer can collect the running speed of the map building device 20 through the four-wheel encoder; in addition, all the above-mentioned acquired data can be transmitted to the server 102 (i.e., industrial control computer) in real time for subsequent processing. Optionally, the image acquisition device 202 can be a vision camera.

[0077] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0078] It is worth mentioning that, in the embodiments of this application, the server 102 is used to acquire radar point cloud data and image acquisition data collected by the lidar and image acquisition device from the area to be tested, respectively. Then, the radar point cloud data is analyzed to obtain the first measurement pose of the map building device in the standard coordinate reference system, and the image acquisition data is analyzed to obtain the second measurement pose of the map building device in the standard coordinate reference system. Based on the first measurement pose and the second measurement pose, the target fusion pose of the map building device is obtained, and then a map of the area to be tested is constructed based on the target fusion pose, the radar point cloud data and the image acquisition data.

[0079] The map construction method provided by the exemplary embodiments of this application will be described below in conjunction with the above system architecture and with reference to the accompanying drawings. It should be noted that the above system architecture is only shown for the purpose of understanding the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0080] See Figure 3 The diagram shown is an implementation flowchart of a map construction method provided in this application embodiment. Taking a server as an example, the specific implementation flow of this method is as follows:

[0081] S301: Acquire radar point cloud data and image acquisition data from the area to be tested.

[0082] It should be noted that both radar point cloud data and image acquisition data contain environmental information collected by the map building device during its patrol of the area under test at the current sampling time. Among them, radar point cloud data is used to indicate the target point cloud position of each target (e.g., pedestrians, trees, etc.) in the radar coordinate system in the area under test acquired by lidar, and image acquisition data is used to indicate the target image position of each target (e.g., pedestrians, trees, etc.) in the image coordinate system in the area under test acquired by image acquisition device.

[0083] In one alternative implementation, after obtaining radar point cloud data and image acquisition data, the server can perform data preprocessing, such as removing stray points from the radar point cloud data and image acquisition data, to reduce the impact of irrelevant data on subsequent map construction.

[0084] S302: Analyze the radar point cloud data to obtain the first measurement pose of the map building device in the standard coordinate reference system, and analyze the image acquisition data to obtain the second measurement pose of the map building device in the standard coordinate reference system.

[0085] The standard coordinate reference system is also known as the world coordinate reference system or the absolute coordinate reference system. Therefore, in the process of analyzing radar point cloud data and image acquisition data, it is necessary to perform corresponding coordinate transformations, that is, to unify the obtained radar point cloud data and image acquisition data into the standard coordinate system, so as to facilitate subsequent data fusion and map construction.

[0086] For example, during step S302, after the server unifies both the radar point cloud data and image acquisition data to a standard coordinate reference system, it can perform point cloud matching on the radar point cloud data acquired by the lidar through the lidar SLAM front end, thereby obtaining the first measurement pose of the map building device in the standard coordinate reference system (i.e., lidar odometry information), denoted as: X l =(xl ,y l ,θ l ,v xl ,v yl ,w l ), where x l y l θ l v xl v yl w l These respectively represent the horizontal coordinate, vertical coordinate, angle, lateral velocity, vertical velocity, and angular velocity of the map-building device obtained by the LiDAR in the standard coordinate reference system.

[0087] Furthermore, feature extraction and feature matching can be performed on the image acquisition data collected by the image acquisition device through the visual SLAM front-end to obtain the pose transformation between adjacent image frames. Then, through pose motion estimation, the second measurement pose of the map building device in the standard coordinate reference system can be obtained, denoted as X. v =(x v ,y v ,θ v ,v xv ,v yv ,w v ), where x v y v θ v v xv v yv w v These respectively represent the horizontal coordinate, vertical coordinate, angle, horizontal velocity, vertical velocity, and angular velocity of the map building device obtained by the image acquisition device in the standard coordinate reference system.

[0088] S303: Based on the first and second measurement poses, obtain the target fusion pose of the map building device.

[0089] S304: Construct a map of the area to be tested based on target fusion pose, radar point cloud data, and image acquisition data.

[0090] Based on the map construction method steps S301 to S304 above, please refer to... Figure 4As shown, this is a schematic diagram of a specific application scenario of a map construction method provided in this application embodiment. The server acquires the LiDAR Radar1 and the image acquisition device Video1 respectively, at the current sampling time (e.g., 2022.10.18). (13:11:28) Radar point cloud data Radar.Data and image acquisition data Image.Data are collected from the area to be tested, Tar.Area. The radar point cloud data Radar.Data is then parsed to obtain the first measurement pose No.1.Pose of the map building device in the standard coordinate reference system, and the image acquisition data Image.Data is parsed to obtain the second measurement pose No.2.Pose of the map building device in the standard coordinate reference system. Based on the first measurement pose No.1.Pose and the second measurement pose No.2.Pose, the target fusion pose Tar.Fus.Pose of the map building device is obtained. Then, based on the target fusion pose Tar.Fus.Pose, the radar point cloud data Radar.Data, and the image acquisition data Image.Data, a map of the area to be tested, Tar.Area, is constructed, thus obtaining the map of the area to be tested, Tar.Area.

[0091] In an optional embodiment, when performing step S303, after the server obtains the first and second measurement poses of the map building device, it can use the capacitive Kalman filter algorithm to predict the first and second measurement poses respectively to obtain the corresponding first target state information set and second target state information set. The first target state information set and the second target state information set include at least: the target state information matrix and the target state information vector at the current sampling time.

[0092] For example, assume the initial state of the sensor (LiDAR or image acquisition device) is x. k|k , where x k|k The initial covariance matrix can be either the first or second measurement pose described above. k|k The state (prediction) equation is x t =f(x) t )+Q t The observation equation is z t =h(x t )+v t , where Q t and v t If the noise consists of state noise and observation noise, then the capacitive Kalman filter algorithm is as follows:

[0093] 1. State prediction.

[0094] (1) Calculate the volume point.

[0095]

[0096]

[0097] in, P represents the predicted volumetric sampling point obtained from the sensor at time k, where 2n represents the number of volumetric sampling points. k|k S represents the covariance matrix of the sensor obtained from the prediction at time k. k|k Represents the covariance matrix P k|k The corresponding arithmetic square root (i.e., standard deviation) matrix, ξ j Represents an n×2n dimensional matrix. This represents the predicted target state value obtained by the sensor at time k.

[0098] (2) Propagate volume points and calculate one-step prediction.

[0099]

[0100]

[0101] in, Let f(*) represent the predicted volumetric sampling point of the sensor at time k+1, and let f(*) represent the nonlinear state transition function of the target. This represents the predicted target state value obtained by the sensor at time k+1.

[0102] (3) Calculate the state error covariance matrix in one step.

[0103]

[0104] Among them, P k+1|k Let Q represent the covariance matrix of the sensor obtained from the prediction at time k+1. k Let represent the covariance matrix of the state noise during the target's motion at time k.

[0105] (4) Calculate the state information and Fisher information, namely the target state information matrix and the target state information vector.

[0106]

[0107]

[0108] Among them, Y k+1|k This represents the target state information matrix obtained by the sensor at time k+1, as predicted. P represents the covariance matrix obtained from the prediction. k+1|k The inverse matrix, y k+1|kThis represents the target state information vector obtained by the sensor at time k+1, as predicted. This represents the predicted target state value obtained by the sensor at time k+1.

[0109] 2. Measurement update.

[0110] (1) Calculate the volume point.

[0111]

[0112]

[0113] Among them, P k+1|k S represents the covariance matrix of the sensor at time k+1 obtained from the measurement. k+1|k This represents the standard deviation matrix of the sensor measured at time k+1. S represents the standard deviation matrix k+1|k The transpose of the matrix, ξ represents the j-th volumetric sampling point of the sensor at time k, obtained through measurement. j Represents an n×2n dimensional matrix. This represents the predicted target state value obtained by the sensor at time k+1.

[0114] (2) Propagate volume points and calculate one-step prediction.

[0115]

[0116]

[0117] in, Let represent the j-th volumetric sampling point of the sensor at time k+1, and h(*) represent the nonlinear measurement transfer function of the target. This represents the target state prediction value obtained by the sensor at time k+1.

[0118] (3) Calculate the covariance matrix.

[0119]

[0120] in, This represents the cross-covariance matrix of the sensor's predicted and measured values ​​at time k+1. This represents the predicted volumetric sampling point of the sensor at time k+1. This represents the predicted target state value obtained by the sensor at time k+1. This represents the j-th volumetric sampling point of the sensor at time k+1, obtained through measurement. This represents the target state prediction value obtained by the sensor at time k+1.

[0121] (4) Calculate the information contribution matrix and information contribution vector, namely the fusion association information matrix and the fusion state information increment.

[0122] According to the capacitive Kalman filter algorithm: Then we have:

[0123]

[0124]

[0125] in, P represents the cross-covariance matrix of the predicted and measured values ​​of the sensor at time k+1 after correction. k+1|k+1 This represents the corrected covariance matrix at time k+1. H represents the pseudo-measurement matrix H of the sensor at time k+1. k+1 The transpose of I k+1 This represents the fused correlation information matrix at time k+1. R represents the covariance matrix R of the sensor's measurement noise at time k+1. k+1 The inverse matrix, i, is the fused state information increment. k+1 V represents the increment of the fused state information at time k+1. k+1 Let represent the covariance matrix of the observation noise during the target's motion at time k+1.

[0126] (5) Update the information state and Fisher state, namely the target state information matrix and the target state information vector.

[0127] Y k+1|k+1 =Y k+1|k +I k+1

[0128]

[0129] Among them, Y k+1|k+1 Y represents the corrected target state information matrix of the sensor at time k+1. k+1|k I represents the target state information matrix obtained by the sensor at time k+1, which is the result of prediction. k+1 This represents the fused correlation information matrix at time k+1. Let y represent the corrected target state information vector of the sensor at time k+1. k+1|k Let i represent the target state information vector obtained by the sensor at time k+1, where i is the target state information vector obtained by the prediction. k+1 This represents the increment of the fused state information at time k+1.

[0130] 3. Status update.

[0131]

[0132]

[0133] Among them, P k+1|k+1 This represents the corrected covariance matrix at time k+1. Y represents the target state information matrix of the sensor at time k+1 after correction. k+1|k+1 The inverse matrix, This represents the corrected target state prediction value of the sensor at time k+1, i.e., the corrected pose of the map-building device. This represents the corrected target state information vector of the sensor at time k+1.

[0134] It should be noted that if there is data from N sensors to be fused, the initial state of the s-th sensor is... The observation equations for each sensor are as follows: If s = 1, 2, ..., N, then the corresponding update information state and Fisher state, i.e., the target state information matrix and the target state information vector, can be represented as follows:

[0135]

[0136]

[0137] Therefore, substituting the above expression into the state update formula in step 3 yields the multi-sensor fusion state.

[0138] Therefore, based on the above-mentioned capacitive Kalman filter algorithm, after the server obtains the first target state information set and the second target state information set, it can obtain the target fusion pose of the map building device according to the first target state information set and the second target state information set.

[0139] For details, please refer to Figure 5 As shown, the server obtains the fusion state information increment and fusion association information matrix at the current sampling time based on the first state information increment and the first association information matrix contained in the first target state information set, and the second state information increment and the second association information matrix contained in the second target state information set. Thus, based on the target state information matrix, the target state information vector, the fusion state information increment and the fusion association information matrix, the target fusion pose of the map building device is obtained.

[0140] It should be noted that after the server obtains the target fusion pose of the map building device, it can perform backend optimization of visual SLAM to further reduce the estimation error in the calculation process. For example, in the local map, bundle adjustment is used for local map optimization, and in the global map, graph optimization is used for global map optimization, further reducing the estimation error in the map building process.

[0141] In one alternative embodiment, see [reference] Figure 6 As shown, during step S304, after obtaining the target fusion pose, the server can obtain the corresponding map requirement type from the map construction request sent by the receiving business requester. Then, based on the map construction method set according to the corresponding map requirement type, the radar point cloud data and image acquisition data are adjusted to obtain the adjusted radar point cloud data and image acquisition data. Finally, based on the target fusion pose and the adjusted radar point cloud data and image acquisition data, a map of the area to be tested is constructed.

[0142] For example, considering the high real-time requirements and accurate obstacle estimation during navigation, the server can carry the corresponding map requirement type according to the map building request, and combine radar point cloud data and image acquisition data to generate a 2D grid map for navigation. That is, since a single LiDAR can only acquire environmental information within the radar scanning plane, which is insufficient to meet the needs of accurate navigation, the 3D point cloud map built by visual SLAM is projected into a 2D grid map, and then fused with the 2D grid map built by LiDAR SLAM based on weighted Bayesian method. Finally, a high-precision fused navigation map is obtained, that is, a map of the area to be tested is constructed.

[0143] Optionally, the server can also generate a 3D point cloud map with better visualization effects based on the corresponding map requirements and by combining radar point cloud data and image acquisition data. In particular, aligning the 3D point cloud map obtained by visual SLAM with the 2D grid map obtained by laser SLAM can yield a more accurate 3D visualized point cloud map.

[0144] In one optional implementation, during the execution of step S301, in order to improve the positioning and map building accuracy of the map building device in situations such as high-speed movement and turning, the server can also acquire the third and fourth measurement poses of the map building device collected from the area to be measured by the inertial measurement unit and the wheel odometer at the current sampling time, respectively. Based on the first, second, third, and fourth measurement poses, the target fusion pose of the map building device is obtained.

[0145] For example, the server calculates the inertial measurement unit (IMU) pose by processing the acceleration and angular velocity information collected by the IMU, thereby obtaining the inertial measurement pose of the map building device in the measurement coordinate reference system of the IMU. It also calculates the wheel odometer information based on the pulse count of the four-wheel encoder, which is the wheel measurement pose of the map building device in the wheel coordinate reference system corresponding to the wheel odometer. Then, it performs the corresponding coordinate transformation to unify the inertial measurement pose and the wheel measurement pose into the standard coordinate system, thereby obtaining the corresponding third and fourth measurement poses for subsequent data fusion and map building.

[0146] In one alternative implementation, after obtaining the third and fourth measurement poses, the server can perform data fusion on the third and fourth measurement poses based on a preset Kalman filter algorithm, such as an extended Kalman filter algorithm, to obtain the corresponding initial fused pose.

[0147] For example, suppose the server obtains a third measurement pose corresponding to the inertial measurement unit, denoted as X. l =(x i ,y i ,θ i ,v xi ,v yi ,w i ), where x i y i θ i v xi v yi w i These respectively represent: the x-coordinate, y-coordinate, angle, lateral velocity, longitudinal velocity, and angular velocity of the map-building device obtained by the inertial measurement unit in the standard coordinate reference system, as well as the fourth measurement pose corresponding to the wheel odometer, denoted as X. o =(x o ,y o ,θ o ,v xo ,v yo ,w o ), where x o y o θ o v xo v yo w o These represent the x-coordinate, y-coordinate, angle, lateral velocity, longitudinal velocity, and angular velocity of the map-building device obtained by the wheeled odometer in the standard coordinate reference system. An extended Kalman filter algorithm can then be used to determine the third measurement pose X. l The pose is fused with the fourth measurement pose to obtain the corresponding initial fused pose, which can be denoted as X. init=(x init ,y init ,θ init ,v xinit ,v yinit ,w init ), where x init y init θ init v xinit v yinit and w init These represent the initial fused abscissa, initial fused ordinate, initial fused angle, initial fused lateral velocity, initial fused longitudinal velocity, and initial fused angular velocity of the map building device in the standard coordinate reference system.

[0148] It should be noted that if the current sampling time is the first sampling time, the initial fused pose X obtained above can be used. init As the initial pose for visual SLAM and laser SLAM.

[0149] Therefore, after obtaining the initial fused pose, the server can use a similar method, namely, employing a capacitive Kalman filter algorithm, to predict the initial fused pose, the first measurement pose, and the second measurement pose, respectively, to obtain the corresponding initial fused state information set, the first target state information set, and the second target state information set. Then, based on these sets, the target fused pose of the map building device can be obtained, i.e., a second pose fusion is performed. The corresponding map requirement type is then obtained from the map building request sent by the business requester. Next, based on the map building method set according to the corresponding map requirement type, the radar point cloud data and image acquisition data are adjusted to obtain adjusted radar point cloud data and image acquisition data. Finally, based on the target fused pose and the adjusted radar point cloud data and image acquisition data, a map of the area to be measured is constructed to improve the positioning and map building accuracy of the map building device in situations such as high-speed movement and turning.

[0150] It should also be noted that the above-mentioned methods for constructing maps using various sensors in this application embodiment are merely examples, and no restrictions are placed on the applicable scope or usage scenarios of the above-mentioned map construction methods. In addition, the initial fused pose can also be processed in other ways before being fused with the first measurement pose and the second measurement pose. Therefore, in this application embodiment, no specific limitations are made on the pose fusion with the first measurement pose and the second measurement pose after the initial fused pose is introduced.

[0151] In summary, the map building method provided in this application embodiment acquires radar point cloud data and image acquisition data from the area to be measured, then analyzes the radar point cloud data to obtain the first measurement pose of the map building device in the standard coordinate reference system, and analyzes the image acquisition data to obtain the second measurement pose of the map building device in the standard coordinate reference system. Based on the first and second measurement poses, the target fusion pose of the map building device is obtained, and then a map of the area to be measured is constructed based on the target fusion pose, radar point cloud data, and image acquisition data.

[0152] This approach avoids the problem in related technologies that fail to consider that the maps built by each sensor are based on the pose of their respective map-building devices. Therefore, it achieves tightly coupled map building with multiple sensors and improves the accuracy of map building.

[0153] Furthermore, based on the same technical concept, embodiments of this application provide a map building apparatus for implementing the above-described method flow of embodiments of this application. See also... Figure 7 As shown, the map building device includes: an acquisition module 701, a parsing module 702, a fusion module 703, and a building module 704, wherein:

[0154] The acquisition module 701 is used to acquire radar point cloud data and image acquisition data from the area to be tested;

[0155] The parsing module 702 is used to parse radar point cloud data to obtain the first measurement pose of the map building device in the standard coordinate reference system, and to parse image acquisition data to obtain the second measurement pose of the map building device in the standard coordinate reference system.

[0156] The fusion module 703 is used to obtain the target fusion pose of the map building device based on the first measurement pose and the second measurement pose;

[0157] Module 704 is used to construct a map of the area to be tested based on target fusion pose, radar point cloud data and image acquisition data.

[0158] In an optional embodiment, when obtaining the target fusion pose of the map building device based on the first and second measurement poses, the fusion module 703 is specifically used for:

[0159] The capacitive Kalman filter algorithm is used to predict the first and second measurement poses respectively, and obtain the corresponding first target state information set and second target state information set; based on the first and second target state information sets, the target fusion pose of the map building device is obtained.

[0160] In one optional embodiment, the first target state information set includes: a target state information matrix, a target state information vector, a first state information increment, and a first association information matrix; the second target state information set includes: a target state information matrix, a target state information vector, a second state information increment, and a second association information matrix.

[0161] When obtaining the target fusion pose of the map building device based on the first target state information set and the second target state information set, the fusion module 703 is specifically used for:

[0162] Based on the first state information increment and the second state information increment, the corresponding fused state information increment is obtained, and based on the first association information matrix and the second association information matrix, the corresponding fused association information matrix is ​​obtained.

[0163] Based on the target state information matrix, target state information vector, fusion state information increment, and fusion association information matrix, the target fusion pose of the map building device is obtained.

[0164] In one embodiment, during the process of acquiring collected radar point cloud data and image acquisition data from the area to be tested, the acquisition module 701 is further configured to:

[0165] The third and fourth measurement poses of the map building device, respectively, are acquired from the area to be measured by the inertial measurement unit and the wheeled odometer.

[0166] In an optional embodiment, when obtaining the target fusion pose of the map building device based on the first target state information set and the second target state information set, the fusion module 703 is further configured to:

[0167] Based on the preset Kalman filter algorithm, the third and fourth measurement poses are fused to obtain the corresponding initial fused pose.

[0168] The capacitive Kalman filter algorithm is used to predict the initial fusion pose and obtain the corresponding initial fusion state information set.

[0169] Based on the initial fusion state information set, the first target state information set, and the second target state information set, the target fusion pose of the map building device is obtained.

[0170] In an optional embodiment, when constructing a map of the area to be measured based on target fusion pose, radar point cloud data, and image acquisition data, the construction module 704 is specifically used for:

[0171] Obtain the corresponding map requirement type from the map building request;

[0172] Based on the map construction method set according to the corresponding map requirement type, the radar point cloud data and image acquisition data are adjusted to obtain the adjusted radar point cloud data and image acquisition data.

[0173] A map of the area to be measured is constructed based on the target fusion pose and adjusted radar point cloud data and image acquisition data.

[0174] Based on the same technical concept, embodiments of this application also provide an electronic device that can implement the map construction method flow provided in the above embodiments of this application. In one embodiment, the electronic device may be a server, a terminal device, or other electronic devices. Figure 8 As shown, the electronic device may include:

[0175] At least one processor 801 and a memory 802 connected to at least one processor 801. In this embodiment, the specific connection medium between the processor 801 and the memory 802 is not limited. Figure 8 The example shown is the connection between processor 801 and memory 802 via bus 800. Bus 800 is... Figure 8 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 800 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 8 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 801 can also be called a controller; there is no restriction on the name.

[0176] In this embodiment, memory 802 stores instructions executable by at least one processor 801. By executing the instructions stored in memory 802, at least one processor 801 can execute a map construction method as described above. Processor 801 can implement... Figure 7 The functions of each module in the device shown.

[0177] The processor 801 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 802 and calling data stored in memory 802, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0178] In an alternative design, processor 801 may include one or more processing units. Processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 801. In some embodiments, processor 801 and memory 802 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0179] The processor 801 can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the map construction method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0180] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 802 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 802 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0181] By designing and programming the processor 801, the code corresponding to the map construction method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute it during runtime. Figure 3The illustrated embodiment presents the steps of a map construction method. How to design and program the processor 801 is a technique well-known to those skilled in the art and will not be described further here.

[0182] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a map construction method described above.

[0183] In some alternative embodiments, this application also provides that various aspects of a map building method can also be implemented as a program product including program code, which, when the program product is run on a device, causes the control device to perform the steps in a map building method according to various exemplary embodiments of this application as described above.

[0184] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0185] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0186] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0188] Program code for performing the operations of this application can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0189] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents...

Claims

1. A map construction method, characterized in that, Applied to map building devices, including: From the area to be tested, acquire radar point cloud data and image acquisition data, and acquire the third and fourth measurement poses of the map building device acquired from the area to be tested by the inertial measurement unit and the wheel odometer, respectively. The radar point cloud data is analyzed to obtain the first measurement pose of the map building device in the standard coordinate reference system; The image acquisition data is parsed to obtain the second measurement pose of the map building device in the standard coordinate reference system; The capacitive Kalman filter algorithm is used to predict the first measurement pose and the second measurement pose respectively, so as to obtain the corresponding first target state information set and second target state information set; Based on a preset Kalman filter algorithm, the third measurement pose and the fourth measurement pose are fused to obtain the corresponding initial fused pose; The capacitive Kalman filter algorithm is used to predict the initial fused pose to obtain the corresponding initial fused state information set. Based on the initial fusion state information set, the first target state information set, and the second target state information set, the target fusion pose of the map building device is obtained; A map of the area to be tested is constructed based on the target fusion pose, the radar point cloud data, and the image acquisition data.

2. The method as described in claim 1, characterized in that, The first target state information set includes: a target state information matrix, a target state information vector, a first state information increment, and a first association information matrix; the second target state information set includes: the target state information matrix, the target state information vector, a second state information increment, and a second association information matrix. Based on the first target state information set and the second target state information set, the target fusion pose of the map building device is obtained, including: Based on the first state information increment and the second state information increment, the corresponding fused state information increment is obtained, and based on the first association information matrix and the second association information matrix, the corresponding fused association information matrix is ​​obtained. Based on the target state information matrix, the target state information vector, the fusion state information increment, and the fusion association information matrix, the target fusion pose of the map building device is determined.

3. The method according to any one of claims 1-2, characterized in that, The process of constructing a map of the area to be measured based on the target fused pose, the radar point cloud data, and the image acquisition data includes: Obtain the corresponding map requirement type from the map building request; Based on the map construction method set according to the map requirement type, the radar point cloud data and the image acquisition data are adjusted to obtain the adjusted radar point cloud data and image acquisition data. Based on the target fusion pose and the adjusted radar point cloud data and image acquisition data, a map of the area to be tested is constructed.

4. A map building device, characterized in that, Applied to map building devices, including: The acquisition module is used to acquire radar point cloud data and image acquisition data collected from the area to be tested, and to acquire the third and fourth measurement poses of the map building device collected from the area to be tested by the inertial measurement unit and the wheel odometer, respectively. The parsing module is used to parse the radar point cloud data to obtain the first measurement pose of the map building device in the standard coordinate reference system, and to parse the image acquisition data to obtain the second measurement pose of the map building device in the standard coordinate reference system. The fusion module is used to predict the first measurement pose and the second measurement pose respectively using the capacitive Kalman filter algorithm to obtain the corresponding first target state information set and second target state information set; Based on a preset Kalman filter algorithm, the third measurement pose and the fourth measurement pose are fused to obtain the corresponding initial fused pose; The capacitive Kalman filter algorithm is used to predict the initial fused pose to obtain the corresponding initial fused state information set. Based on the initial fusion state information set, the first target state information set, and the second target state information set, the target fusion pose of the map building device is obtained; A construction module is used to construct a map of the area to be tested based on the target fusion pose, the radar point cloud data, and the image acquisition data.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-3.

7. A computer program product, characterized in that, When the computer program product is invoked by a computer, it causes the computer to perform the method as described in any one of claims 1-3.

Images