A method, device, equipment and storage medium for constructing three-dimensional map

Abstract

The present invention discloses a method, device, equipment and storage medium for constructing a three-dimensional map. The method comprises: controlling a robot to call a laser radar to collect a first point cloud for a specified area, and calling a binocular camera to collect a second point cloud for the area at the same time, wherein the binocular camera is an RGB-D camera; calculating a deviation between the first point cloud and the second point cloud; if the deviation is greater than a preset first threshold, constructing a three-dimensional map for the area according to the first point cloud; if the deviation is less than or equal to the preset first threshold, fusing the first point cloud and the second point cloud into a third point cloud, and constructing a three-dimensional map for the area according to the third point cloud, thereby improving the accuracy of the three-dimensional map and being able to adapt well to different environmental conditions while ensuring the accuracy of the three-dimensional map.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and particularly to a method, device, equipment and storage medium for constructing a three-dimensional map. Background Art

[0002] Currently, Simultaneous Localization and Mapping (SLAM) is an important link in the field of robot navigation. A robot is equipped with multiple sensors to establish a global map of the explored environment and can, at any time, infer its own position using this map. Among them, the data characteristics of the sensors and the correlation of the observation data directly affect the accuracy of the robot's positioning and map construction.

[0003] During the construction process in a closed cable trench, the difficulty of manual wire laying is high and there are safety risks. A wire laying robot can be used to replace manual work to complete the wire laying. The point cloud collected by lidar is relatively sparse, and some information will be missing in the three-dimensional map established with the sparse point cloud. In addition, the working environment of the wire laying robot is relatively complex, with situations such as low light, low contrast, and obstacles at different heights. When using binocular vision to collect point clouds, it is greatly affected by the environment. When the environmental conditions are not met, the deviation of the point clouds collected by binocular vision is large, and the established three-dimensional map will also have a large corresponding error. Summary of the Invention

[0004] The present invention provides a method, device, equipment and storage medium for constructing a three-dimensional map to solve the problem that it is difficult to adapt to various environments and establish an accurate three-dimensional map due to the sparse point cloud collected by lidar and the large influence of binocular cameras on the environment.

[0005] According to one aspect of the present invention, there is provided a method for constructing a three-dimensional map, the method comprising:

[0006] Controlling the robot to call a lidar to collect a first point cloud for a specified area, and simultaneously calling a binocular camera to collect a second point cloud for the area, the binocular camera being an RGB-D camera;

[0007] Calculating the deviation between the first point cloud and the second point cloud;

[0008] If the deviation is greater than a preset first threshold, constructing a three-dimensional map for the area according to the first point cloud;

[0009] If the deviation is less than or equal to the preset first threshold, fusing the first point cloud and the second point cloud into a third point cloud, and constructing a three-dimensional map for the area according to the third point cloud.

[0010] According to another aspect of the present invention, there is provided a device for constructing a three-dimensional map, the device comprising:

[0011] A point cloud acquisition module, configured to control a robot to call a lidar to acquire a first point cloud for a specified area, and at the same time call a binocular camera to acquire a second point cloud for the area, where the binocular camera is an RGB-D camera;

[0012] A deviation calculation module, configured to calculate the deviation between the first point cloud and the second point cloud;

[0013] A three-dimensional map construction module, configured to, if the deviation is greater than a preset first threshold, construct a three-dimensional map for the area according to the first point cloud; if the deviation is less than or equal to the preset first threshold, fuse the first point cloud and the second point cloud into a third point cloud, and construct a three-dimensional map for the area according to the third point cloud.

[0014] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for constructing a three-dimensional map according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the method for constructing a three-dimensional map according to any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention adds the acquisition of the second point cloud by a binocular camera on the basis of the acquisition of the first point cloud by a lidar. Since the point cloud acquired by the lidar has the characteristics of being stable but sparse, the confidence of the first point cloud is high but the amount of information is not much. When the deviation between the first point cloud and the second point cloud is large, it means that the error of the second point cloud acquired by the binocular camera is large. At this time, only the first point cloud can be used to construct a three-dimensional map. When the deviation between the first point cloud and the second point cloud is small, it means that the confidence of the second point cloud is high, and there are differences in the information contained in the point clouds acquired by different sensors. The second point cloud with high confidence can be used as a supplement to the sparse first point cloud. The fusion result obtained by fusing the first point cloud and the second point cloud maintains rich information and high confidence. Using the fusion result to construct a three-dimensional map can improve the accuracy of the three-dimensional map, and can well adapt to different environmental conditions while ensuring the accuracy of the three-dimensional map.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 is a flowchart of a method for constructing a three-dimensional map according to Embodiment 1 of the present invention;

[0023] Figure 2 is a schematic diagram of the coordinate system of a camera model according to Embodiment 1 of the present invention;

[0024] Figure 3 is a schematic diagram of the joint calibration of a binocular camera and a lidar according to Embodiment 1 of the present invention;

[0025] Figure 4 is a schematic diagram of a point cloud fusion process according to Embodiment 1 of the present invention;

[0026] Figure 5 is a schematic diagram of the structure of a device for constructing a three-dimensional map according to Embodiment 2 of the present invention;

[0027] Figure 6 is a schematic diagram of the structure of an electronic device for implementing the method for constructing a three-dimensional map of the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present invention are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 The figure is a flowchart of a method for constructing a three-dimensional map provided in Embodiment 1 of the present invention. This embodiment is applicable to a complex environment with poor lighting conditions such as a cable trench for a wire laying robot. This method can be executed by a three-dimensional map construction device, and the three-dimensional map construction device can be implemented in the form of hardware and / or software.

[0032] Traditional stereo vision-based methods are sensitive to lighting conditions and feature extraction is incomplete. In the case of low contrast, the embodiments of the present invention propose a measurement system that combines stereo vision information with a three-dimensional lidar and estimates the pose and speed of the target through the EKF algorithm. When the binocular vision matching condition is not met, the pose and speed of the target can be estimated based on the three-dimensional information of the lidar; conversely, when the three-dimensional reconstruction condition is met, the stereo vision reconstructed point cloud information is merged with the acquired point cloud information of the lidar to obtain the target point cloud data. Further, the EKF algorithm is used to estimate the pose and simulate the movement of a non-cooperative target. By fusing the binocular vision system and the three-dimensional lidar system, the measurement scheme can meet the requirements for both short and long distances in space. In addition, the system can also adapt to sudden changes in the spatial lighting conditions and provide a pose measurement and motion estimation method with good robustness and high estimation accuracy for the target.

[0033] As Figure 1 shown, the method includes:

[0034] S110. Control the robot to call the lidar to collect the first point cloud of a specified area, and at the same time call the binocular camera to collect the second point cloud of the area. The binocular camera is an RGB-D camera.

[0035] In this step, the vision system has advantages such as independence, accuracy, reliability, and information integrity, and can perform functions such as target recognition, obstacle avoidance, and path planning. Since monocular vision cannot obtain the depth information of the target object and it is difficult to estimate the three-dimensional position information, in order to meet the navigation requirements in complex environments, a stereo vision navigation method is introduced into the field of autonomous navigation. In the embodiments of the present invention, the binocular camera used is an RGB-D camera (depth camera). The RGB-D camera adds a depth measurement to the ordinary camera function, and the output depth map is an image containing information related to the surface distance of the scene object from the viewpoint. The depth image is a common three-dimensional scene representation method. Two cameras with fixed relative positions can be used to obtain the three-dimensional information of the carrier, but the measurement method of the vision system is easily affected by the lighting conditions and fails, especially when the target feature extraction is incomplete and the contrast is low.

[0036] The lidar has good stability, but the point cloud is sparse and there are some deficiencies when precise identification and capture points are required.

[0037] Establish a lidar and a stereo camera model according to the size and parameters of the lidar and the binocular camera. In the embodiments of the present invention, the lidar uses a single-line lidar RPLIDAR A2 (a laser scanning ranging radar) lidar system. The system supports 16 channels in the horizontal field of view, the measurement angle range is 360°, the vertical field of view is 30°, ±15° up and down, and the effective range is 100m; the depth camera can use a specified model. Exemplarily, the depth camera of the specified model can obtain the depth map through optical coding technology, and its effective range of depth ranging is 0.6 - 8 meters, with an accuracy of 3 millimeters, and its depth camera viewing angle can reach 58° horizontally and 45.5° vertically.

[0038] Among them, the two-dimensional laser collects data in the horizontal 360° range, and the odometer provides the initial value of the scan matching, which can well calculate the pose during rapid rotation and make up for the defect that the depth camera's viewing angle is insufficient and cannot handle rapid rotation.

[0039] The left and right cameras independently identify any corresponding point P i (x i , y i , z i ) of the target. The pixel coordinates of this point on the image planes of the two cameras can be expressed as p L (u L , v L ). The camera coordinate systems of the left camera and the right camera are respectively represented as O L -X L Y L Z L and O R -XR Y R Z R , and the coordinates of any point in the lidar body coordinate system (such as O VLP -X VLP Y VLP Z VLP can be expressed as (x VLP , y VLP , z VLP ).

[0040] In the embodiments of the present invention, by way of example, for the application requirements of the cable laying robot for obstacle detection and environmental mapping in the cable trench, due to the defects of the single-sensor measurement method in terms of accuracy and stability, combined with the complexity of the environment, the traditional vision-based measurement method is prone to failure due to the influence of lighting conditions; although lidar has good stability, the point cloud is sparse and there are some deficiencies when precise identification and capture of points are required. Using lidar and a binocular camera to jointly map a specified area can well adapt to environmental changes.

[0041] In specific implementation, a lidar model needs to be established. A single-line lidar can be horizontally installed on the robot and rotated 360° through a pan-tilt head to emit laser light around. After the laser light encounters an obstacle and returns, it is acquired by the receiver, and the distance to the obstacle is calculated by calculating the time difference between emission and reception. When ignoring the sampling time difference at different angles during the laser rotation, the observation model of the 3D lidar can be expressed by the conditional probability distribution as follows:

[0042] p(z k |x k , m)

[0043] where x k is the pose of the robot at time k, m is the map information (the map information includes a list of object information in the environment and their positions), z k is the laser observation. The laser is sampled multiple times per rotation, that is

[0044]

[0045] it can be considered that all laser points in a single scan are independent of each other. Then for a complete circle of scans, there is:

[0046]

[0047] In addition, a stereo camera model needs to be established. When establishing the stereo camera model, since the binocular camera is a depth camera RGB-D, it can simultaneously acquire color images and depth images. First, model the camera, such as Figure 2As shown in the coordinate system schematic diagram of the camera model, from the similarity relationship of triangles, the point P[X c Y c Z c T in the coordinate system of the camera model and its imaging point [X p ,Y p ,f] T on the physical plane xoy have the following relationship:

[0048]

[0049] where f is the imaging focal length of the camera.

[0050] The relationship between the imaging point [X p ,Y p ,f] T on the physical plane xoy and the pixel coordinates [u v] T is as follows:

[0051]

[0052] where α and β are the magnification coefficients in the x and y directions, with the unit of pixel / meter, and x c ,y c are the coordinate offsets of the camera optical center on the image.

[0053] Finally, the relationship between the obtained pixel coordinates and the three-dimensional coordinates in the camera coordinate system is:

[0054]

[0055] where is the internal parameter matrix of the camera.

[0056] Calibrate the lidar and camera according to the fusion calibration algorithm. Referring to Figure 3 the schematic diagram of the joint calibration of the binocular camera and the lidar, first perform the joint calibration of the lidar and the camera. The purpose of the joint calibration is to obtain the homogeneous transformation matrix between the lidar coordinate system and the camera coordinate system, so that the lidar data corresponds to the camera image.

[0057] The image data collected by the left camera is represented by (u L ,v L ), and the three-dimensional point cloud data collected by the lidar is represented by (x VLP ,y VLP ,z VLP ). The goal of calibration is to create a transformation matrix M that maps the three-dimensional points (x, y, z) to the 2D points (u L ,v L ), that is: ​

[0058]

[0059] Among them, is the internal parameter matrix of the left camera; f uL , f vL represents the scale factor in the xy-axis directions (i.e., the effective focal lengths in the horizontal and vertical directions); u L0 , v L0 are the principal coordinate system of the image plane; M L = [R L t L , R L is a 3×3 attitude rotation matrix, and t L is a 3×1 translation vector.

[0060] On the other hand, according to the imaging principle of the left camera, it can be obtained that:

[0061]

[0062] Then the formula is obtained as follows:

[0063]

[0064] Therefore, the rotation matrix and translation vector of the lidar coordinate system relative to the left camera coordinate system are:

[0065]

[0066] For the convenience of mechanical installation and better target recognition, the lidar can be installed in the middle of the binocular camera. The calibration board consists of 9 rows and 12 columns, each square being 25 mm, and the camera is about 2 m away from the calibration board. The detailed joint calibration steps are as follows:

[0067] (1) Adjust the camera. Adjust the position of the camera according to factors such as distance, f (focal length), etc., so that the calibration board is placed within the fields of view of the camera and the lidar. To obtain better calibration results, the proportion of the calibration board in the field of view should be as high as possible.

[0068] (2) Obtain images. At a distance of about 2 m from the calibration board, the binocular camera captures 20 images of different positions and angles of the calibration board through external triggering.

[0069] (3) Calculate the internal and external parameters of the binocular camera. Use the Matlab calibration toolbox to calculate the internal and external parameters and distortion coefficients of the binocular camera.

[0070] (4) Obtain lidar point cloud data. Extract the three-dimensional coordinates of each corner point of the lidar (corresponding to the pixel coordinates of the binocular camera).

[0071] (5) Calculate the external parameters of the measurement system. According to the calibration results of the binocular camera, the external parameters c of the measurement system are obtained from the formula to obtain the external parameter c of the measurement system L R VLP and C L t VLP .

[0072] Through model establishment and joint calibration, the following calibration parameters can be obtained:

[0073] The internal parameter correction matrix and the parameters of the distorted binocular camera are respectively:

[0074]

[0075]

[0076] The external parameter matrix of the binocular camera is:

[0077]

[0078]

[0079] Based on the joint calibration algorithm, the calibration result of the external parameter three-dimensional lidar coordinate system is determined. For the left camera frame, the following results can be obtained:

[0080]

[0081] c L t VLP = [-202.48 1.0698 -19.41] T (mm)

[0082] In addition, coordinate transformation can be performed on the right camera in the same way.

[0083] S120. Calculate the deviation between the first point cloud and the second point cloud.

[0084] In this step, when the deviation between the first point cloud and the second point cloud is large, it can be inferred that there is a high probability that the second point cloud collected by the binocular camera is not accurate enough, that is, the environmental conditions are not sufficient for the binocular camera to collect accurate point cloud information; while when the deviation between the first point cloud and the second point cloud is small, it can be inferred that the second point cloud collected by the binocular camera is more accurate.

[0085] In one embodiment, S120 includes the following steps:

[0086] S120-1. Calculate the first center of the first point cloud;

[0087] S120-2. Calculate the second center of the second point cloud;

[0088] S120 - 3, calculate the distance between the first center and the second center as the deviation between the first point cloud and the second point cloud.

[0089] In this step, after obtaining the point cloud data of the binocular camera and the lidar simultaneously, the first center and the second center can be calculated. Among them, the first center can be the mean value of all point cloud coordinates in the first point cloud, and the second center can be the mean value of all point cloud coordinates in the second point cloud. The distance between the first center and the second center can be the Euclidean distance calculated by the three - dimensional space formula, and this distance is used as the deviation between the first point cloud and the second point cloud.

[0090] S130. If the deviation is greater than the preset first threshold, construct a three - dimensional map for the area according to the first point cloud.

[0091] In this step, a large distance indicates a large deviation between the first point cloud and the second point cloud. It is speculated that the data collected by the binocular camera may be inaccurate, while the lidar has good stability and is not easily affected by the environment to cause large point cloud errors. At this time, only the first point cloud collected by the lidar can be used to construct a three - dimensional map for the area. Although there will be a situation of fewer point clouds when constructing a three - dimensional map only with the first point cloud, the overall effect will be more accurate than the three - dimensional map constructed by combining the second point cloud with larger errors.

[0092] S140. If the deviation is less than or equal to the preset first threshold, fuse the first point cloud and the second point cloud into a third point cloud, and construct a three - dimensional map for the area according to the third point cloud.

[0093] In this step, a small distance indicates a small deviation between the first point cloud and the second point cloud. It is speculated that the data collected by the binocular camera has a high accuracy, and the on - site environment can meet the conditions for the binocular camera to obtain accurate point cloud information. The first point cloud and the second point cloud can be fused to obtain the third point cloud after fusion. That is, the binocular stereo vision information and the lidar information can be combined for fusion measurement. The non - cooperative target is three - dimensionally reconstructed by the binocular stereo vision system and the three - dimensional lidar system respectively, and the data of the two obtained point clouds is fused to obtain a complete three - dimensional point cloud image.

[0094] In one embodiment, for fusing the first point cloud and the second point cloud into a third point cloud in S140, the following steps are included:

[0095] S141, project the first point cloud and the second point cloud onto the same plane.

[0096] In this step, during the fusion process of the first point cloud and the second point cloud, the first point cloud and the second point cloud collected by the calibrated binocular camera and lidar respectively can be projected onto the same plane.

[0097] In one embodiment, the coordinate system of the first point cloud is the laser coordinate system, the coordinate system of the second point cloud is the camera coordinate system, and the second point cloud includes ground point cloud; S140-1 includes the following steps:

[0098] Convert the point cloud in the laser coordinate system to the camera coordinate system;

[0099] Construct a plane equation containing the origin of the camera coordinate system through plane fitting;

[0100] Identify the ground point cloud from the second point cloud and remove the ground point cloud from the second point cloud;

[0101] Project the first point cloud after coordinate transformation and the second point cloud after removing the ground point cloud onto the plane corresponding to the plane equation.

[0102] In this step, since the first point cloud collected by the lidar and the second point cloud collected by the binocular camera are not in the same coordinate system, using the rotation and translation matrix between the lidar and the binocular camera obtained by joint calibration, the point clouds obtained by both can be converted to the same coordinate system. First, convert the two-dimensional point cloud in the laser coordinate system to the camera coordinate system and construct a plane equation containing the origin of the camera coordinate system through plane fitting: ax + by + cz = d, that is, the fitted plane equation. Since the second point cloud collected by the binocular camera very likely contains ground point cloud, and the ground point cloud does not belong to the obstacle information during robot navigation, this part of the point cloud needs to be removed. Finally, by ignoring the Z-axis information of the three-dimensional point cloud describing spatial obstacles obtained by the binocular camera and only retaining the X-axis and Y-axis information, the three-dimensional point cloud can be projected onto the plane corresponding to the plane equation ax + by + cz = d.

[0103] S142, perform a rigid transformation on the first point cloud until the first point cloud is aligned with the second point cloud.

[0104] In this step, assume that [x tk y tk z tk T and [x tk+1 y tk+1 z tk+1 T respectively represent the three-dimensional coordinates obtained by the fusion measurement system at time t k and t k+1 moments. Therefore, the rigid transformation between matching points at adjacent times satisfies the following relationship, that is:

[0105]

[0106] where, Δ k R k+1 and Δ​​k t k+1 respectively represent the rotation matrix and translation vector of the target.

[0107] S143. For the aligned first point cloud and second point cloud, calculate the error between the first point cloud and the second point cloud based on the relationship between the lidar and the binocular camera.

[0108] In this step, the relationship between the lidar and the binocular camera can refer to the transformation relationship between the point cloud coordinates obtained after joint calibration. The error between the first point cloud and the second point cloud can be used to determine whether the first point cloud and the second point cloud are successfully matched.

[0109] In one embodiment, S143 includes the following steps:

[0110] For the aligned first point cloud and second point cloud, calculate the geometric features of the first point cloud and the geometric features of the second point cloud. The geometric features include the average distance δ, curvature ρ, and normal angle φ.

[0111] Calculate the error between the first point cloud and the second point cloud through the following formula:

[0112]

[0113] where E is the error between the first point cloud and the second point cloud, N m is the number of points in the first point cloud, N d is the number of points in the second point cloud, P k and P k+1 are a pair of corresponding point clouds, ω i，j is the weighting coefficient of the corresponding point clouds, m i is the i-th point in the first point cloud, d j is the j-th point in the second point cloud, ΔR is the rotation matrix between m i and d j and Δt is the translation vector between m i and d j N is the total number of points in the first point cloud and the second point cloud, || || and || || are both norm operation symbols, and the corresponding point clouds are a pair of point clouds that are projected onto the same plane and indicate the same location.

[0114] In this step, the corresponding point clouds refer to a pair of point clouds that are projected onto the same plane and indicate the same location. By introducing the geometric features of the first point cloud and the second point cloud into the error function, the obtained error result is the matching error between the first point cloud and the second point cloud, which can improve the accuracy of point cloud registration.

[0115] S144. If the error is less than a preset second threshold, multiply the coordinates of the first point cloud by a first weight to obtain a first result, multiply the coordinates of the second point cloud by a second weight to obtain a second result, calculate the sum of the first result and the second result as the coordinates of the third point cloud, and use the coordinates of the third point cloud as the fusion result. Among them, if the accuracy of the lidar is higher than that of the binocular camera, the first weight is greater than the second weight; if the accuracy of the lidar is lower than that of the binocular camera, the first weight is less than the second weight.

[0116] In this step, if the error is less than the preset second threshold, it means that the matching between the first point cloud and the second point cloud is completed, and the third point cloud can be solved to achieve the fusion of the first point cloud and the second point cloud. The first weight and the second weight can be set by the user according to the situation of the device. Exemplarily, the user can allocate the weights according to the accuracy level.

[0117] S145. If the error is greater than or equal to the preset second threshold, adjust the relationship between the lidar and the binocular camera, and return to execute the calculation of the error between the first point cloud and the second point cloud based on the relationship between the lidar and the binocular camera for the aligned first point cloud and second point cloud.

[0118] In this step, if the error is greater than or equal to the preset second threshold, it means that the first point cloud and the second point cloud are not completely matched. The matching can be redone by minimizing the formula for calculating the error. Optimizing the formula for calculating the error can be expressed as:

[0119]

[0120] That is, after inputting the geometric features, a new rotation matrix R and translation vector t are obtained by minimizing the error, achieving the effect of optimizing the error formula. After adjusting the relationship between the lidar and the binocular camera to obtain the optimized formula for calculating the error, it is possible to return to the aligned first point cloud and second point cloud, calculate the error between the first point cloud and the second point cloud based on the relationship between the lidar and the binocular camera, perform matching and error calculation again until the matching is completed and then achieve the fusion of the first point cloud and the second point cloud.

[0121] In one embodiment, for constructing a three-dimensional map of the area according to the third point cloud in S140, the following steps are included:

[0122] Perform motion estimation on the lidar to obtain a first motion result, and perform motion estimation on the binocular camera to obtain a second motion result;

[0123] Perform pose estimation on the lidar to obtain a first pose, and perform pose estimation on the binocular camera to obtain a second pose;

[0124] Determine the first target pose based on the first pose and the first motion result;

[0125] Determine the second target pose based on the second pose and the second motion result;

[0126] Use a Kalman filter to fuse the first target pose and the second target pose to obtain a fused pose;

[0127] Construct a three-dimensional map of the area based on the fused pose.

[0128] In this step, after obtaining the fused third point cloud, attitude measurement and motion estimation can be started based on the third point cloud. To estimate the attitude and speed, two coordinate systems are defined in the embodiments of the present invention, namely the reference coordinate system and the target coordinate system. The reference coordinate system is set in the left camera coordinate system, and the coordinate origin is located at the center of the left camera, that is, the reference coordinate system can be expressed as The target coordinate system is O b -X b Y b Z b , and its coordinate origin is located at the rotation center of the target. The motion estimation method can be divided into the following steps:

[0129] (1) According to the three-dimensional point cloud model of the non-cooperative target, calculate the geometric features δ, ρ, φ, etc. of the target, and identify the target.

[0130] (2) According to the EKF (Extended Kalman Filter) algorithm, perform ICP matching (Iterative Closest Point, an algorithm based on data registration method, using the nearest point search method to solve an algorithm based on free-form surfaces) on two adjacent point cloud models to obtain the attitude and speed of the non-cooperative tumbling target. The extended Kalman filter (EKF) is as follows:

[0131] The Kalman filter is applicable to linear Gaussian systems and is used to obtain the optimal estimate of the system under the optimal estimate with the minimum mean square error. The extended Kalman filter extends the condition that the Kalman filter must be in a linear Gaussian system. By performing a Taylor expansion near the filtering point and linearizing the system using the first-order approximation value, it can be applied to the non-linear system in the embodiments of the present invention. For the fusion part of the visual and laser positioning results in the embodiments of the present invention, the extended Kalman filter method is used for pose fusion.

[0132] For non-linear systems,

[0133]

[0134] where u k is the control input, w k and v kThey are the process noise and the observation noise respectively. Both of them are multivariate Gaussian noises with zero mean, and their covariances are Q k , R k . In the formula, the function f can calculate the prediction of the state at the next moment from the state at the previous moment, and h is used to calculate the predicted observation from the predicted state. It is not necessarily a linear function of the state, but it must be non-linearly differentiable.

[0135] In the extended Kalman filter, after linearizing the system at the current moment, the Jacobian at each moment is used to predict the state at the next moment.

[0136] The extended Kalman filter is divided into two parts: prediction and update.

[0137] The prediction part is as follows:

[0138] x k|k-1 = f(x k-1|k-1 , u k )

[0139]

[0140] The update part is as follows:

[0141] y k = z k - h(x k|k-1 )

[0142]

[0143] x k|k = x k|k-1 + K k y k

[0144] P k|k = (I - K k H k )P k|k-1

[0145] Among them, the elements of the state transition matrix F k and the elements of the Jacobian matrix H of the observation equation k are respectively

[0146]

[0147]

[0148] In the prediction stage, since the role of the observation is not considered, the state and its covariance are predicted respectively:

[0149] Through the posterior x at the previous moment k-1|k-1 and the current control input u kPredict the prior x at the current moment k|k-1 ; P k|k-1 is the prediction of the covariance prior.

[0150] In the update phase, y k is the process of the observation residual S k ; calculate the approximate Kalman gain K k , that is, related to the weight of the residual in the update; finally, correct the prediction according to the observation to obtain the updated state estimate x k|k ; P k|k is the updated covariance posterior estimate.

[0151] Visual feature points are used for motion estimation after matching, and the laser method uses correlation matching for motion estimation. Therefore, when visual and laser positioning are both successful at the same time, the system outputs two poses simultaneously, and the EKF fusion is performed on the two pose results; when visual tracking is unsuccessful, the positioning result of the laser is used to splice the point cloud data of the depth camera to obtain a three-dimensional map. At the same time, continue to detect and match features in subsequent frames, re-initialize the map points in visual SLAM. If successful, continue to use the fusion mode of the lidar and the binocular camera, otherwise, always use the positioning result of the laser to build a three-dimensional map.

[0152] The positioning information obtained by visual SLAM is a three-dimensional motion with six degrees of freedom. When fusing with the two-dimensional pose obtained by the lidar, it is necessary to decompose its motion on the two-dimensional map plane, that is, decompose the pose components on the XY plane in the world coordinate system from the three-dimensional rotation matrix representing the camera pose. Since the RGB-D camera and the lidar in the embodiments of the present invention are both horizontally installed, it is considered that the pose change on the ZX plane in the camera coordinate system is the XY pose in the world coordinate system. Then the problem is converted into an extended Kalman filter fusion problem of two two-dimensional motions. The following problems need to be noted in the application:

[0153] (1) Over time, cumulative errors will occur in the absolute poses obtained from visual and laser SLAM, so the relative pose differences of each sensor are used to update the extended Kalman filter.

[0154] (2) When the robot moves, its uncertainty in the world reference becomes larger and larger. Over time, the covariance will grow infinitely. Therefore, the covariance of the published pose is invalid, and the covariance of the velocity needs to be published.

[0155] (3) Since the sensor data does not arrive at exactly the same time, it is necessary to interpolate the sensor data during filter fusion.

[0156] When visual SLAM tracking and positioning fails and a relocalization strategy is adopted, the data provided by laser SLAM can be used to continuously obtain the pose and then provided to the visual end. At the same time, if the collected visual scene can be re-initialized, the positioning data of laser SLAM is used to restart tracking to obtain an uninterrupted positioning result. When both visual and laser positioning are successful, the extended Kalman filter is used to fuse the positioning results. An octree map can be built through visual SLAM to avoid the shortcoming of laser SLAM that can only build maps for two-dimensional planes, which can be used for obstacle avoidance.

[0157] In one embodiment, after S110, it further includes:

[0158] Denoise the first point cloud and the second point cloud respectively.

[0159] In this step, after collecting the first point cloud and the second point cloud, the first point cloud obtained by the lidar can be denoised, and points with large jumps can be removed according to a preset threshold to obtain an effective first point cloud; in addition, a filtering algorithm can be used to denoise the binocular image, feature extraction and feature matching are performed on the collected image, the RANSAC algorithm is used to eliminate mismatched feature points from the set of matched feature points, and the three-dimensional coordinates of the spatial point set are reconstructed by the least squares method to obtain the second point cloud.

[0160] In addition, the embodiment of the present invention uses a two-wheel differential drive robot chassis Turtlebot2 with an open interaction interface. The relative motion between two samplings of the robot can be obtained through the open interaction interface; the software runs on the Ubuntu16.04 operating system and uses the corresponding version Kinetic of the Robot Operating System (ROS), and commonly used third-party libraries such as the Open Source Computer Vision Library (OpenCV), General Graph Optimization (g2o), Pointcloud (PCL), and Octomap are used for basic data processing and display.

[0161] To enable those skilled in the art to have a better understanding of the embodiments of the present invention, the following example is used to further explain the fusion of the first point cloud and the second point cloud:

[0162] Reference Figure 4Schematic diagram of the point cloud fusion process. First, by matching the geometric features of two point clouds \(P_k\) and \(P_{k + 1}\), such as the average distance \(\delta\), curvature \(\rho\), and normal angle \(\varphi\). Here, the two point clouds can be point clouds at two adjacent moments. Calculate the geometric parameters and corresponding relationships between points in the point clouds at adjacent moments, and eliminate the jump points in the point cloud in combination with the corresponding relationships. After obtaining the geometric parameters between points, through rigid transformation, find the corresponding matching points, and calculate the matching error between the two point sets according to the error function based on these geometric features. If the matching error range is lower than the threshold, the matching is completed; otherwise, by minimizing the error function, re-match and calculate the new rigid transformation. Finally, determine the target point cloud and output the target point cloud.

[0163] The technical solution of the embodiment of the present invention adds the acquisition of the second point cloud by the binocular camera on the basis of the lidar acquiring the first point cloud. Since the point cloud acquired by the lidar has the characteristics of being stable but sparse, the confidence of the first point cloud is high but the amount of information is not much. When the deviation between the first point cloud and the second point cloud is large, it means that the error of the second point cloud acquired by the binocular camera is large. At this time, only the first point cloud can be used to construct the 3D map. When the deviation between the first point cloud and the second point cloud is small, it means that the confidence of the second point cloud is high. And for the information contained in the point clouds acquired by different sensors, there are differences. The second point cloud with high confidence can be used as a supplement to the sparse first point cloud. The fusion result obtained by fusing the first point cloud and the second point cloud maintains rich information and high confidence. Using the fusion result to construct the 3D map can improve the accuracy of the 3D map and can well adapt to different environmental conditions while ensuring the accuracy of the 3D map.

[0164] Embodiment 2

[0165] Figure 5 It is a schematic structural diagram of a 3D map construction device provided by Embodiment 2 of the present invention. As Figure 5 shown, the device includes:

[0166] A point cloud acquisition module 510, configured to control the robot to call the lidar to acquire the first point cloud for a specified area, and at the same time call the binocular camera to acquire the second point cloud for the area, where the binocular camera is an RGB-D camera;

[0167] A deviation calculation module 520, configured to calculate the deviation between the first point cloud and the second point cloud;

[0168] A 3D map construction module 530, configured to, if the deviation is greater than a preset first threshold, construct a 3D map for the area according to the first point cloud; if the deviation is less than or equal to the preset first threshold, fuse the first point cloud and the second point cloud into a third point cloud, and construct a 3D map for the area according to the third point cloud.

[0169] In one embodiment, the deviation calculation module 520 includes the following sub-modules:

[0170] The first center calculation sub-module is used to calculate the first center of the first point cloud;

[0171] The second center calculation sub-module is used to calculate the second center of the second point cloud;

[0172] The distance calculation sub-module is used to calculate the distance between the first center and the second center as the deviation between the first point cloud and the second point cloud.

[0173] In one embodiment, for fusing the first point cloud and the second point cloud into a third point cloud in the three-dimensional map construction module 530, the following sub-modules are included:

[0174] The projection sub-module is used to project the first point cloud and the second point cloud onto the same plane;

[0175] The rigid transformation sub-module is used to perform a rigid transformation on the first point cloud until the first point cloud is aligned with the second point cloud;

[0176] The error calculation sub-module is used to calculate the error between the first point cloud and the second point cloud for the aligned first point cloud and second point cloud based on the relationship between the lidar and the binocular camera;

[0177] The execution sub-module is used to, when the error is less than a preset second threshold, multiply the coordinates of the first point cloud by a first weight to obtain a first result, multiply the coordinates of the second point cloud by a second weight to obtain a second result, calculate the sum of the first result and the second result as the coordinates of the third point cloud, and use the coordinates of the third point cloud as the fusion result. Wherein, if the accuracy of the lidar is higher than the accuracy of the binocular camera, the first weight is greater than the second weight; if the accuracy of the lidar is lower than the accuracy of the binocular camera, the first weight is less than the second weight; when the error is greater than or equal to the preset second threshold, adjust the relationship between the lidar and the binocular camera, and return to execute calculating the error between the first point cloud and the second point cloud for the aligned first point cloud and second point cloud based on the relationship between the lidar and the binocular camera.

[0178] In one embodiment, the error calculation sub-module includes the following units:

[0179] The geometric feature calculation unit is used to calculate the geometric features of the first point cloud and the geometric features of the second point cloud for the aligned first point cloud and second point cloud, and the geometric features include the average distance δ, the curvature ρ, and the normal angle φ;

[0180] An error calculation unit for calculating the error between the first point cloud and the second point cloud through the following formula:

[0181]

[0182] where E is the error between the first point cloud and the second point cloud, N m is the number of point clouds in the first point cloud, N d is the number of point clouds in the second point cloud, P k and P k+1 are a pair of corresponding point clouds, ω i，j is the weighting coefficient of the corresponding point clouds, m i is the i-th point in the first point cloud, d j is the j-th point in the second point cloud, ΔR is m i and d j is the rotation matrix between and d i and d j is the translation vector between and d, N is the total number of point clouds of the first point cloud and the second point cloud, |||| and || || are both norm operation symbols, and the corresponding point clouds are a pair of point clouds that the first point cloud and the second point cloud are projected onto the same plane and indicate the same place.

[0183] In one embodiment, the coordinate system where the first point cloud is located is a laser coordinate system, the coordinate system where the second point cloud is located is a camera coordinate system, and the second point cloud includes ground point clouds;

[0184] The projection sub-module includes the following units:

[0185] A coordinate conversion unit for converting the point cloud in the laser coordinate system to the camera coordinate system;

[0186] A plane equation construction unit for constructing a plane equation including the origin of the camera coordinate system through plane fitting;

[0187] A ground point cloud clearing unit for identifying the ground point cloud from the second point cloud and clearing the ground point cloud from the second point cloud;

[0188] A projection unit for projecting the first point cloud after coordinate conversion and the second point cloud after clearing the ground point cloud onto the plane corresponding to the plane equation.

[0189] In one embodiment, constructing a three-dimensional map of the area for the third point cloud in the three-dimensional map construction module 530 includes the following sub-modules:

[0190] A motion result estimation sub-module, configured to perform motion estimation on the lidar to obtain a first motion result, and perform motion estimation on the binocular camera to obtain a second motion result;

[0191] A pose estimation sub-module, configured to perform pose estimation on the lidar to obtain a first pose, and perform pose estimation on the binocular camera to obtain a second pose;

[0192] A first target pose determination sub-module, configured to determine a first target pose according to the first pose and the first motion result;

[0193] A second target pose determination sub-module, configured to determine a second target pose according to the second pose and the second motion result;

[0194] A fused pose determination sub-module, configured to fuse the first target pose and the second target pose by using a Kalman filter to obtain a fused pose;

[0195] A three-dimensional map construction sub-module, configured to construct a three-dimensional map of the area based on the fused pose.

[0196] In one embodiment, the following modules are further included:

[0197] A denoising module, configured to perform denoising processing on the first point cloud and the second point cloud respectively.

[0198] The three-dimensional map construction device provided by the embodiment of the present invention can execute the three-dimensional map construction method provided by the first embodiment of the present invention, and has corresponding function modules and beneficial effects for executing the method.

[0199] Embodiment III

[0200] Figure 6 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described herein and / or claimed.

[0201] As Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0202] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0203] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for constructing a three-dimensional map.

[0204] In some embodiments, the method for constructing a three-dimensional map can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for constructing a three-dimensional map described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the method for constructing a three-dimensional map by any other appropriate means (e.g., by means of firmware).

[0205] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0206] A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0207] In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. A more specific example of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0208] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0209] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0210] The computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0211] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0212] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for constructing a three-dimensional map, characterized in that: The method comprises: Control the robot to call the laser radar to collect a first point cloud for a specified area, and at the same time call the binocular camera to collect a second point cloud for the area, wherein the binocular camera is an RGB-D camera; Calculating a deviation between the first point cloud and the second point cloud; If the deviation is greater than a preset first threshold, constructing a three-dimensional map of the area according to the first point cloud; If the deviation is less than or equal to a preset first threshold, fusing the first point cloud with the second point cloud into a third point cloud, and constructing a three-dimensional map of the area according to the third point cloud; The calculating the deviation between the first point cloud and the second point cloud comprises: Calculating a first center of the first point cloud; Calculating a second center of the second point cloud; Calculate the distance between the first center and the second center as the deviation between the first point cloud and the second point cloud; wherein the first center is the mean of all point cloud coordinates in the first point cloud, and the second center is the mean of all point cloud coordinates in the second point cloud; the distance between the first center and the second center is the Euclidean distance calculated by a three-dimensional space formula.

2. The method according to claim 1, characterized in that: The step of fusing the first point cloud and the second point cloud into a third point cloud comprises: Projecting the first point cloud and the second point cloud onto the same plane; Performing a rigid transformation on the first point cloud until the first point cloud is aligned with the second point cloud; For the aligned first point cloud and the second point cloud, calculating an error between the first point cloud and the second point cloud based on a relationship between the laser radar and the binocular camera; If the error is less than a preset second threshold, the coordinates of the first point cloud are multiplied by a first weight to obtain a first result, the coordinates of the second point cloud are multiplied by a second weight to obtain a second result, the sum of the first result and the second result is calculated as the coordinates of the third point cloud, and the coordinates of the third point cloud are used as the fusion result, wherein if the accuracy of the laser radar is higher than the accuracy of the binocular camera, the first weight is greater than the second weight, and if the accuracy of the laser radar is lower than the accuracy of the binocular camera, the first weight is less than the second weight; If the error is greater than or equal to a preset second threshold, the relationship between the laser radar and the binocular camera is adjusted, and the process returns to executing the aligned first point cloud and the second point cloud, and the error between the first point cloud and the second point cloud is calculated based on the relationship between the laser radar and the binocular camera.

3. The method according to claim 2, characterized in that The step of calculating the error between the first point cloud and the second point cloud that have been aligned based on the relationship between the laser radar and the binocular camera includes: For the aligned first point cloud and the second point cloud, calculating geometric features of the first point cloud and the second point cloud, the geometric features comprising an average distance δ, a curvature ρ, and a normal angle φ; The error between the first point cloud and the second point cloud is calculated by the following formula: Where E is the error between the first point cloud and the second point cloud, N m is the number of point clouds in the first point cloud, N d is the number of point clouds in the second point cloud, P k and P k+1 is a pair of point clouds, ω i，j is the weight coefficient of paired point clouds, m i is the i-th point in the first point cloud, d j is the jth point in the second point cloud, ΔR is m i With d j The rotation matrix between Δt and m i With d j The translation vector between them, N is the total number of point clouds of the first point cloud and the second point cloud, |||| and |||| are both norm operation symbols, and the paired point cloud is a pair of point clouds indicating the same place when the first point cloud and the second point cloud are projected onto the same plane.

4. The method according to claim 2, characterized in that: The coordinate system where the first point cloud is located is a laser coordinate system, the coordinate system where the second point cloud is located is a camera coordinate system, and the second point cloud includes a ground point cloud; The projecting the first point cloud and the second point cloud onto the same plane includes: Convert the point cloud in the laser coordinate system to the camera coordinate system; Constructing a plane equation including the origin of the camera coordinate system by plane fitting; Identifying a ground point cloud from the second point cloud, and removing the ground point cloud from the second point cloud; The first point cloud after the coordinate system is converted and the second point cloud after the ground point cloud is cleared are projected onto the plane corresponding to the plane equation.

5. The method according to any one of claims 1 to 4, characterized in that: The constructing a three-dimensional map of the area according to the third point cloud comprises: Performing motion estimation on the laser radar to obtain a first motion result, and performing motion estimation on the binocular camera to obtain a second motion result; Performing pose estimation on the laser radar to obtain a first pose, and performing pose estimation on the binocular camera to obtain a second pose; Determine a first target posture according to the first posture and the first movement result; Determine a second target posture according to the second posture and the second movement result; Using a Kalman filter to fuse the first target posture and the second target posture to obtain a fused posture; Based on the fused pose, a three-dimensional map of the area is constructed.

6. The method according to any one of claims 1 to 4, characterized in that: After the control robot calls the laser radar to collect the first point cloud of the designated area and calls the binocular camera to collect the second point cloud of the area, the method further includes: De-noising is performed on the first point cloud and the second point cloud respectively.

7. A three-dimensional map construction device, characterized in that: The device comprises: A point cloud acquisition module, used to control the robot to call the laser radar to collect a first point cloud for a specified area, and to call the binocular camera to collect a second point cloud for the area, wherein the binocular camera is an RGB-D camera; a deviation calculation module, used to calculate the deviation between the first point cloud and the second point cloud; a three-dimensional map construction module, configured to construct a three-dimensional map of the area according to the first point cloud if the deviation is greater than a preset first threshold; and to fuse the first point cloud with the second point cloud into a third point cloud if the deviation is less than or equal to the preset first threshold, and to construct a three-dimensional map of the area according to the third point cloud; The deviation calculation module comprises: A first center calculation submodule, used to calculate the first center of the first point cloud; A second center calculation submodule, used to calculate the second center of the second point cloud; A distance calculation submodule is used to calculate the distance between the first center and the second center as the deviation between the first point cloud and the second point cloud; wherein the first center is the mean of all point cloud coordinates in the first point cloud, and the second center is the mean of all point cloud coordinates in the second point cloud; the distance between the first center and the second center is the Euclidean distance calculated by a three-dimensional space formula.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for constructing a three-dimensional map according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the three-dimensional map construction method described in any one of claims 1 to 6 when executed by a processor.

Images