Point Cloud Map Construction Method, Device, Equipment, and Storage Medium

Abstract

Embodiments of the present application disclose a point cloud map construction method, apparatus, device, and storage medium. Among them, the method includes: performing camera internal parameter calibration on the actual size of the calibration code in the calibration board in the physical space and at least three calibration board images in the sample image sequence to obtain the camera internal parameters; determining the first camera pose of the calibration board image according to the camera internal parameters and the calibration board image; performing map reconstruction using the sample image sequence to obtain a scale-free point cloud map and the second camera pose of each sample image; and performing scale recovery on the scale-free point cloud map according to the first camera pose and the second camera pose corresponding to each calibration board image to obtain a scaled point cloud map.

Description

Technical Field

[0001] The embodiments of the present application relate to computer vision technology, including but not limited to methods and devices, equipment, and storage media for point cloud map construction. Background Art

[0002] Simultaneous Localization And Mapping (SLAM) refers to a subject equipped with specific sensors to build a model of the environment during movement and estimate its own movement without prior knowledge of the environment. If the sensor here is mainly a camera, it is called "Visual SLAM". Visual SLAM has a wide range of applications in fields such as mobile robots, unmanned aerial vehicles, autonomous driving, virtual reality, and augmented reality. Therefore, obtaining a point cloud map closer to the true scale is of great value in these fields. Summary of the Invention

[0003] In view of this, the point cloud map construction methods, devices, equipment, and storage media provided by the embodiments of the present application can obtain a point cloud map closer to the true scale. The point cloud map construction methods, devices, equipment, and storage media provided by the embodiments of the present application are implemented as follows:

[0004] The point cloud map construction method provided by the embodiments of the present application includes: using the actual size of the calibration code in the calibration board in the physical space and at least 3 frames of calibration board images in the sample image sequence to perform camera internal parameter calibration to obtain camera internal parameters; determining the first camera pose of the calibration board image according to the camera internal parameters and the calibration board image; using the sample image sequence to perform map reconstruction to obtain a scale-free point cloud map and the second camera pose of each sample image; and performing scale recovery on the scale-free point cloud map according to the first camera pose and the second camera pose corresponding to each calibration board image to obtain a scaled point cloud map.

[0005] In this way, since a first camera pose closer to the true scale can be obtained based on the camera internal parameters and the calibration board image, a point cloud map closer to the true scale, that is, a scaled point cloud map, can be obtained.

[0006] The point cloud map construction device provided by the embodiment of the present application includes: a camera calibration module, configured to perform camera internal parameter calibration by using the actual size of the calibration code in the calibration board in the physical space and at least 3 frames of calibration board images in the sample image sequence to obtain camera internal parameters; a determination module, configured to determine the first camera pose of the calibration board image according to the camera internal parameters and the calibration board image; a map reconstruction module, configured to perform map reconstruction by using the sample image sequence to obtain a scale-free point cloud map and the second camera pose of each sample image; and a scale recovery module, configured to perform scale recovery on the scale-free point cloud map according to the first camera pose and the second camera pose corresponding to each calibration board image to obtain a scaled point cloud map.

[0007] The electronic device provided by the embodiment of the present application includes a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0008] The computer-readable storage medium provided by the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the method provided by the embodiment of the present application is implemented. Description of the Drawings

[0009] The drawings here are incorporated into the description and form a part of this description. These drawings show embodiments consistent with the present application and are used together with the description to explain the technical solutions of the present application.

[0010] Figure 1 It is a schematic flowchart of the implementation of the point cloud map construction method provided by the embodiment of the present application;

[0011] Figure 2A It is a schematic diagram of the image acquisition scenario of the embodiment of the present application;

[0012] Figure 2B It is a schematic diagram of at least 3 frames of calibration board images described in the embodiment of the present application;

[0013] Figure 3 It is a schematic flowchart of the implementation of another point cloud map construction method provided by the embodiment of the present application;

[0014] Figure 4 It is a schematic flowchart of the implementation of yet another point cloud map construction method provided by the embodiment of the present application;

[0015] Figure 5 It is a schematic flowchart of the implementation of still another point cloud map construction method provided by the embodiment of the present application;

[0016] Figure 6 It is a schematic structural diagram of the point cloud map construction device provided by the embodiment of the present application;

[0017] Figure 7 This is a schematic structural diagram of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will further describe the specific technical solutions of the present application in detail with reference to the accompanying drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] In the following descriptions, "some embodiments" are involved, which describe subsets of all possible embodiments. However, it can be understood that "some embodiments" can be the same subsets or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0021] It should be noted that the terms "first / second / third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0022] To facilitate the understanding of the technical solutions of the embodiments of the present application, the following basic concepts of relevant terms are first given:

[0023] Point cloud map: It is a map represented by a set of discrete sampling points. In this map, at least the three-dimensional coordinates of the sampling points in physical space are included, and it can also carry image features of the sampling points, such as red (Reed, R), green (Green, G), blue (Blue, B) information, and / or feature descriptors, etc. A point cloud map closer to the real scale can better meet the positioning requirements, navigation and obstacle avoidance requirements, as well as visualization and interaction requirements, etc.

[0024] Scale: It refers to that the camera trajectory estimated by monocular SLAM and the point cloud map differ from the real trajectory and map by a factor, that is, the so-called scale (Scale). Since monocular SLAM cannot determine this scale only based on images, it is also called scale ambiguity (Scale Ambiguity).

[0025] Camera pose: It refers to the coordinates and rotation angle of the camera in a specific coordinate system when collecting images. The electronic device can determine the pose of the camera (i.e., the rotation angle) according to the rotation matrix R of the camera coordinate system relative to a specific coordinate system (such as the world coordinate system or a custom coordinate system, etc.), and determine the coordinates of the camera in the specific coordinate system according to the translation matrix T of the camera coordinate system relative to the specific coordinate system.

[0026] Camera intrinsics: It refers to several contents such as the focal length, distortion parameters, and center point of the camera. Camera intrinsics is the key to correcting image distortion. The higher the calibration accuracy of the intrinsics, the better the image distortion correction effect.

[0027] Based on this, an embodiment of the present application provides a method for constructing a point cloud map. This method is applied to an electronic device, which can be various types of devices with information processing capabilities during implementation. For example, the electronic device may include a mobile phone, a tablet computer, a personal computer, a laptop computer, a server, a cluster server, a mobile robot, an unmanned aerial vehicle, or a vehicle-mounted device, etc. The functions implemented by this method can be realized by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0028] Figure 1 It is a schematic diagram of the implementation process of the method for constructing a point cloud map provided by an embodiment of the present application. As Figure 1 shown, this method may include the following steps 101 to step 104:

[0029] Step 101, perform camera intrinsics calibration using the actual size of the calibration code in the calibration board in the physical space and at least 3 frames of calibration board images in the sample image sequence to obtain the camera intrinsics.

[0030] The shape of the calibration code can be a variety of regular geometric figures. For example, the calibration code is a regular polygon such as a rectangle, a square, or a triangle. Correspondingly, the calibration board is composed of calibration codes of different colors at intervals. In some embodiments, the calibration code is a checkerboard calibration code of a specified style, that is, the calibration board is composed of black calibration codes and white calibration codes at intervals; in this way, the camera intrinsics calibration becomes simple and easy to process, thereby reducing the time complexity of the algorithm and further improving the construction speed of the point cloud map.

[0031] In some embodiments, the at least 3 calibration board images are acquired by a camera at different positions along a wavy trajectory perpendicular to the ground, and the overall wavy trajectory is arc-shaped; thus, the at least 3 calibration board images acquired by the camera are calibration board images acquired in the up, down, left, and right directions, that is, these images are taken by the camera at different positions, different angles, and different poses; thus, the universality of the camera internal parameters can be improved, that is to say, for the accurate correction of any image acquired by the camera at any angle, any pose, and any position, the camera internal parameters are applicable.

[0032] For example, as Figure 2A shown, the engineer can first formulate a calibration code: for example, use A4 or A3 paper to print a checkerboard calibration code of a specified style. Since the actual size printed cannot be ensured, the side length of the checkerboard needs to be measured with a ruler in meters and then submitted as a parameter to an electronic device later, such as submitting to a server for offline construction of a point cloud map. Then, the engineer fixes the A4 paper or A3 paper 201 with the checkerboard (i.e., an example of the calibration board) at a position where the camera can observe it well on a predetermined acquisition trajectory; by good observation, it means that the camera can observe the complete content of the calibration board regardless of its position, pose, and angle. Based on this, the preliminary preparation work is completed. As Figure 2A shown, the camera 202 observes the calibration board 201 along the arc-wavy trajectory in the up, down, left, and right directions, thereby obtaining the at least 3 calibration board images; and packs these images and submits them to the server together with the actual size of the checkerboard (such as the side length) for subsequent reconstruction of a scaled point cloud map.

[0033] In the embodiments of the present application, the number of images for camera internal parameter calibration is not limited, and can be 3 frames, or 4 frames, 5 frames, ······, or 20 frames, or even more frames. Theoretically speaking, 3 calibration board images can calibrate accurate camera internal parameters. However, in actual applications, using more than 20 calibration board images can calibrate higher-precision camera internal parameters.

[0034] For example, assume that 22 calibration board images are acquired. As Figure 2B shown, calibration board image 1 to calibration board image 22 are respectively taken by the camera at different positions and different angles; based on these acquired images, higher-precision camera internal parameters can be obtained.

[0035] Step 102, determine the first camera pose of the calibration board image according to the camera internal parameters and the calibration board image.

[0036] The method for determining the first camera pose of each of the calibration board images is the same. There are various methods to implement step 102. In some embodiments, the electronic device can implement step 102 through steps 302 to 304 of the following embodiments. In other embodiments, the electronic device can implement step 102 through steps 402 to 406 of the following embodiments. For specific details, refer to the following embodiments and will not be elaborated here.

[0037] Step 103: Use the sample image sequence for map reconstruction to obtain a scale-free point cloud map and the second camera pose of each of the sample images.

[0038] In some embodiments, the electronic device can use the Structure from Motion (SFM) algorithm to process the sample image sequence, thereby obtaining a scale-free point cloud map and the second camera pose of each sample (including the second camera pose of the calibration board image). Among them, the SFM algorithm includes the following: pairwise match a certain number of images, and use the Euclidean distance judgment method to establish the matching relationship between image feature points; eliminate the matching pairs, and the elimination method is to calculate the fundamental matrix using the RANSAC eight-point method, and the matching pairs that do not satisfy the fundamental matrix are selected to be eliminated; after the matching relationship is established, generate a tracking list, where the tracking list refers to the set of image names of the same-name points; eliminate the invalid matches in the tracking list; find the initialization image pair, the purpose is to find the image pair with the largest camera baseline, and use the RANSAC algorithm four-point method to calculate the homography matrix. The matching points that satisfy the homography matrix are called inliers, and the matching points that do not satisfy the homography matrix are called outliers. Find the image pair with the smallest inlier ratio. Find the relative rotation and translation of the initialization image pair. The method is to calculate the essential matrix through the RANSAC eight-point method, and obtain the relative rotation and translation between the image pairs by performing singular value decomposition (SVD) on the essential matrix, so as to obtain the second camera pose of the images of this image pair; calculate the three-dimensional coordinates of the feature points in the initialization image pair through triangulation; repeatedly execute the above steps for other images, and all image relative rotations and translations can be obtained, that is, the second camera pose of each image, as well as the three-dimensional coordinates of the feature points, so as to obtain a scale-free point cloud map.

[0039] Step 104: According to the first camera pose and the second camera pose corresponding to each calibration board image, perform scale recovery on the scale-free point cloud map to obtain a scaled point cloud map.

[0040] In some embodiments, the electronic device can implement step 104 through steps 306 and 307 of the following embodiments. For specific details, refer to the following embodiments and will not be elaborated here.

[0041] In an embodiment of the present application, the internal parameters of a camera are calibrated by using the actual size of the calibration code in the calibration board in the physical space and at least three calibration board images in the sample image sequence to obtain the internal parameters of the camera; the first camera pose of the calibration board image is determined by using the internal parameters of the camera and the calibration board image; map reconstruction is performed by using the sample image sequence to obtain a scale-free point cloud map and the second camera pose of each sample image; and the scale of the scale-free point cloud map is restored according to the first camera pose and the second camera pose corresponding to each calibration board image to obtain a scaled point cloud map. Thus, since the first camera pose closer to the true scale can be obtained based on the internal parameters of the camera and the calibration board image, a point cloud map closer to the true scale, that is, a scaled point cloud map, can be obtained.

[0042] Another embodiment of the present application provides a method for constructing a point cloud map. Figure 3 It is a schematic flow chart of the implementation of the method for constructing a point cloud map according to an embodiment of the present application. As Figure 3 shown, the method may include the following steps 301 to 307:

[0043] Step 301, perform camera internal parameter calibration by using the actual size of the calibration code in the calibration board in the physical space and at least three calibration board images in the sample image sequence to obtain the internal parameters of the camera.

[0044] In some embodiments, the calibration board may further include identification information for uniquely identifying the calibration board. In this way, the crowdsourcing user can apply online according to the mapping location, which is convenient for the backend to perform map fusion. The backend can judge the map scene according to the identification information and thus perform map fusion.

[0045] Step 302, identify the feature points of the calibration code in the calibration board image.

[0046] In some embodiments, the electronic device may detect the corners of the calibration code in the image through a preset corner detection algorithm and use the corners as the feature points of the calibration code. The preset corner detection algorithm can be a variety of algorithms. For example, the algorithm is a Scale-invariant feature transform (SIFT) algorithm, a Harris algorithm, a FAST algorithm, or the like.

[0047] It can be understood that a corner is usually defined as the intersection of two sides. More strictly speaking, the local neighborhood of a corner should have boundaries in different directions in two different regions.

[0048] Step 303, obtain the spatial coordinates of each feature point in the physical space;

[0049] Step 304: Determine the first camera pose of the calibration board image based on the pixel coordinates of each of the feature points in the calibration board image, the spatial coordinates of each of the feature points, and the camera internal parameters.

[0050] In some embodiments, a specific Perspective-n-Point (PnP) algorithm is used to process the pixel coordinates of each of the feature points in the calibration board image, the spatial coordinates of each of the feature points, and the camera internal parameters to obtain the fifth camera pose of the calibration board image; the fifth camera pose of each calibration board image is optimized by the global Bundle Adjustment (BA) method to obtain the first camera pose of each calibration board image; in this way, the obtained first camera pose can be further closer to the camera pose under the true scale, thereby further improving the accuracy of the scaled point cloud map.

[0051] It can be understood that the PnP algorithm is a method for solving the motion from 3D points to 2D point pairs. It describes how to estimate the camera pose when the spatial coordinates of n points in the physical space (i.e., 3D space) and their projection positions are known. There are many methods for solving the PnP problem. For example, the P3P algorithm for estimating the camera pose with 3 point pairs, the Direct Linear Transformation (DLT) algorithm, the Efficient PnP (EPnP) algorithm, or the UPnP algorithm, etc.

[0052] Step 305: Use the sample image sequence for map reconstruction to obtain a scale-free point cloud map and the second camera pose of each of the sample images.

[0053] Step 306: Determine a scale factor based on the first camera pose and the second camera pose corresponding to each of the calibration board images, where the scale factor represents the conversion relationship between the first camera pose and the second camera pose.

[0054] In some embodiments, the electronic device can align the first camera pose and the second camera pose by using the acquisition timestamp of each image, that is, align the acquisition trajectories of the camera close to the true scale and the acquisition trajectories of the scale-free camera; after alignment, use these aligned first camera poses and second camera poses as the observables of a specific function to find the optimal estimated value of the parameters of the specific function; and use the optimal estimated value as the scale factor; where the scale factor represents the conversion relationship between the first camera pose and the second camera pose; in this way, since the optimal estimated value is used as the scale factor, a more accurate scale factor can be obtained, thereby obtaining a point cloud map closer to the true scale, that is, a scaled point cloud map.

[0055] Further, in some embodiments, the electronic device may use the least squares method to solve the scale coefficient.

[0056] Step 307: Use the scale coefficient to perform scale recovery on the coordinates of the sampling points in the scale-free point cloud map to obtain a scaled point cloud map.

[0057] It can be understood that the scale means that the camera trajectory estimated by monocular SLAM differs from the real camera trajectory by a factor, and the point cloud map estimated by monocular SLAM and the actual map differ by a factor. Therefore, in some embodiments, the electronic device may multiply the scale coefficient by the coordinates of the sampling points in the scale-free point cloud map to obtain a scaled point cloud map.

[0058] Map reconstruction requires a large number of images. Generally, the strategy of key frames is used. Every 1m or a specific number of frames, one frame of image is extracted as a key frame, and the key frame sequence enters the subsequent mapping process. In the subsequent mapping, the scale coefficient can be used to perform scale recovery on the coordinate information of the sampling points obtained subsequently.

[0059] Another embodiment of the present application provides a method for constructing a point cloud map. Figure 4 It is a schematic flowchart of the implementation process of the method for constructing a point cloud map according to the embodiment of the present application. As Figure 4 shown, the method may include the following steps 401 to step 410:

[0060] Step 401: Use the actual size of the calibration code in the calibration board in the physical space and at least 3 frames of calibration board images in the sample image sequence to perform camera internal parameter calibration to obtain the camera internal parameters.

[0061] Step 402: Identify the feature points of the calibration code in the calibration board image.

[0062] Step 403: Obtain the spatial coordinates of each feature point in the physical space.

[0063] Step 404: Determine the third camera pose of the calibration board image according to the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera internal parameters.

[0064] In some embodiments, a specific PnP algorithm is used to process the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera internal parameters to obtain the fifth camera pose of the calibration board image; global BA optimization is performed on the fifth camera pose of each calibration board image to obtain the third camera pose of each calibration board image.

[0065] Understandably, visual observations are accompanied by observation errors. Through global optimization means such as bundle adjustment, the errors can be eliminated to obtain a more accurate true camera trajectory or a camera trajectory closer to the true scale, so that the final scaled point cloud map obtained is also closer to the true scale.

[0066] Step 405: Obtain a fourth camera pose output by an Inertial Measurement Unit (IMU) when the camera acquires the calibration board image.

[0067] Step 406: Jointly calculate the third camera pose of the calibration board image and the corresponding fourth camera pose to obtain the first camera pose.

[0068] Understandably, combining the IMU can output a camera position closer to the true scale, but prior knowledge (calibration codes of specific sizes or other observation means) is still required for calibration to obtain a first camera pose with smaller errors.

[0069] In some embodiments, the electronic device can use Visual-Inertial Odometry (VIO) to jointly calculate the third camera pose of the calibration board image and the corresponding fourth camera pose, so as to obtain the first camera pose of the image. In this way, the accuracy of the first camera pose can be further improved, and then the accuracy of the scaled point cloud map can be improved, better solving the scale uncertainty problem of the unscaled point cloud map.

[0070] Step 407: Use the sample image sequence for map reconstruction to obtain an unscaled point cloud map and a second camera pose of each sample image.

[0071] Step 408: Align the first camera pose with the second camera pose according to the acquisition timestamp of the calibration board image.

[0072] Step 409: Use the aligned first camera pose and second camera pose of each calibration board image as observables of a specific function to find the optimal estimated value of the parameters of the specific function; and use the optimal estimated value as the scale coefficient, where the scale coefficient characterizes the conversion relationship between the first camera pose and the second camera pose.

[0073] Thus, since the optimal estimated value is used as the scale coefficient, a more accurate scale coefficient can be obtained, thereby obtaining a point cloud map closer to the true scale, that is, a point cloud map with scale. Further, in some embodiments, the electronic device may use the least squares method, the SVD algorithm, or the Iterative Closest Point (ICP) matching algorithm, etc., to solve the scale coefficient.

[0074] Step 410, using the scale coefficient, perform scale recovery on the coordinates of the sampling points in the scale-free point cloud map to obtain a point cloud map with scale.

[0075] Next, an exemplary application of the embodiments of the present application in a practical application scenario will be described. As Figure 5 shown, the point cloud map construction method may include the following steps 501 to 508:

[0076] Step 501, identify the checkerboard in the sample image sequence (including the at least 3 calibration board images), and extract the inner corner points of the checkerboard (an example of feature points);

[0077] Step 502, use the pixel coordinates of the inner corner points in the calibration board image and the actual size of the checkerboard in the physical space to calibrate the camera internal parameters;

[0078] Step 503, according to the camera internal parameters, the pixel coordinates of the inner corner points in the calibration board image, and the spatial coordinates of the inner corner points, use the PnP algorithm to solve and obtain the fifth camera pose of the calibration board image;

[0079] Step 504, add the fifth camera pose of each calibration board image to the observation sequence, and perform global BA reprojection pose optimization to obtain the camera trajectory at the true scale;

[0080] Step 505, extract the feature points and the feature descriptors of the feature points in the sample image sequence;

[0081] Step 506, according to the feature points and their feature descriptors of each sample image, perform SFM reconstruction to obtain a scale-free camera trajectory and a scale-free sparse map;

[0082] Step 507, according to the camera trajectory at the true scale and the scale-free camera trajectory, use the least squares method to solve the scale coefficient;

[0083] Step 508, use the scale coefficient to perform scale recovery on the scale-free sparse map to obtain a sparse map with the true scale.

[0084] In the embodiments of the present application, (1) since there is no need to obtain the camera internal parameters in advance, the calibration code can be used for internal parameter calibration when the camera takes pictures, so the device adaptability is strong; (2) the production, carrying and deployment of the calibration code are very convenient, so it is convenient to deploy a sparse map with real scale; (3) based on the above-mentioned strong device adaptability and deployment convenience, it is more suitable as a crowdsourcing mapping solution.

[0085] In the embodiments of the present application, using the calibration code as a calibration object can not only obtain the internal parameters of the camera, but also restore the scale of the visual sparse map; performing global BA optimization on the observations of the inner corner points of the calibration code can accurately restore the true trajectory of the camera, thereby restoring the scale coefficient of the visual sparse map.

[0086] In some embodiments, the calibration code can be modified to include ID information, so that the crowdsourcing users can apply online according to the mapping location, which is convenient for the backend to perform map fusion;

[0087] In some embodiments, if the electronic device is equipped with an IMU, it can be guided to use the calibration code for joint calibration of the IMU and the camera to form a VIO system.

[0088] Based on the foregoing embodiments, the embodiments of the present application provide a point cloud map construction device. The device includes each module included and each unit included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0089] Figure 6 It is a schematic structural diagram of the point cloud map construction device in the embodiments of the present application, as Figure 6 shown, the device 60 includes a camera calibration module 601, a determination module 602, a map reconstruction module 603 and a scale recovery module 604, wherein:

[0090] The camera calibration module 601 is used to perform camera internal parameter calibration by using the actual size of the calibration code in the calibration board and at least 3 frames of calibration board images in the sample image sequence to obtain the camera internal parameters;

[0091] The determination module 602 is used to determine the first camera pose of the calibration board image according to the camera internal parameters and the calibration board image;

[0092] The map reconstruction module 603 is used to perform map reconstruction by using the sample image sequence to obtain a scale-free point cloud map and the second camera pose of each sample image;

[0093] The scale recovery module 604 is configured to perform scale recovery on the scale-free point cloud map according to the first camera pose and the second camera pose corresponding to each calibration board image, so as to obtain a scaled point cloud map.

[0094] In some embodiments, the scale recovery module 604 is configured to: determine a scale coefficient according to the first camera pose and the second camera pose corresponding to each calibration board image, where the scale coefficient represents the conversion relationship between the first camera pose and the second camera pose; and use the scale coefficient to perform scale recovery on the coordinates of the sampling points in the scale-free point cloud map, so as to obtain a scaled point cloud map.

[0095] In some embodiments, the scale recovery module 604 is configured to: align the first camera pose and the second camera pose according to the acquisition timestamps of the calibration board images; use the aligned first camera pose and the second camera pose of each calibration board image as the observables of a specific function, and find the optimal estimated value of the parameters of the specific function; and use the optimal estimated value as the scale coefficient.

[0096] In some embodiments, the calibration code is a checkerboard calibration code of a specified style; the at least three calibration board images are acquired at different positions by the camera along a wavy trajectory perpendicular to the ground, and the overall wavy trajectory is arc-shaped.

[0097] In some embodiments, the determination module 602 is configured to: identify the feature points of the calibration code in the calibration board image; obtain the spatial coordinates of each feature point in the physical space; and determine the first camera pose of the calibration board image according to the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera internal parameters.

[0098] In some embodiments, the determination module 602 is configured to: identify the feature points of the calibration code in the calibration board image; obtain the spatial coordinates of each feature point in the physical space; determine the third camera pose of the calibration board image according to the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera internal parameters; obtain the fourth camera pose output by the IMU when the camera acquires the calibration board image; and perform joint calculation on the third camera pose of the calibration board image and the corresponding fourth camera pose to obtain the first camera pose.

[0099] In some embodiments, a determination module 602 is configured to: process the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera internal parameters by using a specific PnP algorithm to obtain the fifth camera pose of the calibration board image; perform global BA optimization on the fifth camera pose of each calibration board image to obtain the camera pose of each calibration board image.

[0100] The description of the above device embodiments is similar to that of the above method embodiments and has similar beneficial effects to the method embodiments. For the technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0101] It should be noted that in the embodiments of the present application Figure 6 The division of the point cloud map construction device shown above for modules is illustrative, merely a logical function division, and there may be other division methods in actual implementation. In addition, in each embodiment of the present application, each functional unit may be integrated in a processing unit, may exist separately physically, or two or more units may be integrated in one unit. The above integrated units may be implemented in the form of hardware, or in the form of software functional units, or in the form of a combination of software and hardware.

[0102] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the related technology, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing an electronic device to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), magnetic disks, or optical discs that can store program codes. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0103] The embodiments of the present application provide an electronic device Figure 7 which is a schematic diagram of the hardware entity of the electronic device according to the embodiments of the present application. As Figure 7 shown, the electronic device 70 includes a memory 701 and a processor 702. The memory 701 stores a computer program that can run on the processor 702, and when the processor 702 executes the program, it implements the steps in the method provided in the above embodiments.

[0104] It should be noted that the memory 701 is configured to store instructions and applications executable by the processor 702, and can also cache data to be processed or already processed by each module in the processor 702 and the electronic device 70 (such as, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (RAM).

[0105] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0106] An embodiment of the present application provides a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the steps in the method provided in the above method embodiment.

[0107] It should be pointed out here that the descriptions of the above storage medium and device embodiments are similar to those of the above method embodiments and have similar beneficial effects to the method embodiments. For the technical details not disclosed in the storage medium, storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0108] It should be understood that the "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" or "in some embodiments" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the order numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments. The above descriptions of each embodiment tend to emphasize the differences between the embodiments, and the same or similar parts can be referred to each other. For the sake of brevity, they will not be repeated herein.

[0109] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, object A and / or object B can represent: object A exists alone, object A and object B exist simultaneously, and object B exists alone. These three situations.

[0110] It should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising such element.

[0111] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described embodiments are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0112] The modules described above as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules; they can be located in one place or distributed to multiple network units; some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional module in the embodiments of this application can be all integrated in a processing unit, or each module can be separately used as a unit, or two or more modules can be integrated in a unit; the above-mentioned integrated modules can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0114] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: various media such as removable storage devices, read-only memory (ROM), magnetic disks or optical discs that can store program codes.

[0115] Alternatively, if the above integrated units of the present application are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence or the part that contributes to the related art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing an electronic device to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media such as removable storage devices, ROMs, magnetic disks, or optical discs that can store program codes.

[0116] The methods disclosed in several method embodiments provided by the present application can be arbitrarily combined without conflict to obtain new method embodiments.

[0117] The features disclosed in several product embodiments provided by the present application can be arbitrarily combined without conflict to obtain new product embodiments.

[0118] The features disclosed in several method or device embodiments provided by the present application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0119] As described above, the above are only the implementation manners of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for constructing a point cloud map, characterized in that, The method includes: Performing camera intrinsic parameter calibration using the actual size of the calibration code in the calibration board in the physical space and at least 3 frames of calibration board images in the sample image sequence to obtain the camera intrinsic parameters; Determining a first camera pose of the calibration board image according to the camera intrinsic parameters and the calibration board image; the first camera pose refers to the coordinates and rotation angle of the camera in a specific coordinate system when collecting the calibration board image; Performing map reconstruction using the sample image sequence to obtain a scale-free point cloud map and a second camera pose of each sample image; the second camera pose refers to the coordinates and rotation angle of the camera in the specific coordinate system when collecting the sample image; the second camera pose of the sample image includes the second camera pose of the calibration board image; Performing scale recovery on the scale-free point cloud map according to the first camera pose and the second camera pose corresponding to each calibration board image to obtain a scaled point cloud map.

2. The method according to claim 1, wherein The performing scale recovery on the scale-free point cloud map according to the first camera pose and the second camera pose corresponding to each calibration board image to obtain a scaled point cloud map includes: Determining a scale coefficient according to the first camera pose and the second camera pose corresponding to each calibration board image, where the scale coefficient represents the conversion relationship between the first camera pose and the second camera pose; Using the scale coefficient to perform scale recovery on the coordinates of the sampling points in the scale-free point cloud map to obtain a scaled point cloud map.

3. The method according to claim 2, characterized in that The determining a scale coefficient according to the first camera pose and the second camera pose corresponding to each calibration board image includes: Aligning the first camera pose and the second camera pose according to the acquisition timestamp of the calibration board image; Taking the aligned first camera pose and the second camera pose of each calibration board image as the observables of a specific function, and finding the optimal estimated value of the parameters of the specific function; and Taking the optimal estimated value as the scale coefficient.

4. The method according to claim 1, wherein The calibration code is a checkerboard calibration code of a specified style; the at least 3 frames of calibration board images are collected at different positions by the camera along a wavy trajectory perpendicular to the ground, and the overall wavy trajectory is arc-shaped.

5. The method according to claim 1, characterized in that The determining a first camera pose of the calibration board image according to the camera intrinsic parameters and the calibration board image includes: Identifying the feature points of the calibration code in the calibration board image; Obtaining the spatial coordinates of each feature point in the physical space; Determining the first camera pose of the calibration board image according to the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera intrinsic parameters.

6. The method according to claim 1, wherein The determining a first camera pose of the calibration board image according to the camera intrinsic parameters and the calibration board image includes: Identifying the feature points of the calibration code in the calibration board image; Obtaining the spatial coordinates of each feature point in the physical space; Determining a third camera pose of the calibration board image according to the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera intrinsic parameters. Obtain the fourth camera pose output by the inertial measurement unit (IMU) when the camera captures the calibration board image; Perform joint calculation on the third camera pose of the calibration board image and the corresponding fourth camera pose to obtain the first camera pose.

7. The method according to claim 5 or 6, characterized in that, Determine the camera pose of the calibration board image according to the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera internal parameters, including: Use a specific PnP algorithm to process the pixel coordinates of each feature point in the calibration board image, the spatial coordinates of each feature point, and the camera internal parameters to obtain the fifth camera pose of the calibration board image; Perform global bundle adjustment (BA) optimization on the fifth camera pose of each calibration board image to obtain the camera pose of each calibration board image.

8. A point cloud map construction device, characterized in that, Including: A camera calibration module for calibrating the camera internal parameters using the actual size of the calibration code in the calibration board in the physical space and at least 3 frames of calibration board images in the sample image sequence to obtain the camera internal parameters; A determination module for determining the first camera pose of the calibration board image according to the camera internal parameters and the calibration board image; the first camera pose refers to the coordinates and rotation angle of the camera in a specific coordinate system when capturing the calibration board image; A map reconstruction module for performing map reconstruction using the sample image sequence to obtain a scale-free point cloud map and the second camera pose of each sample image; the second camera pose refers to the coordinates and rotation angle of the camera in the specific coordinate system when capturing the sample image; the second camera pose of the sample image includes the second camera pose of the calibration board image; A scale recovery module for performing scale recovery on the scale-free point cloud map according to the first camera pose and the second camera pose corresponding to each calibration board image to obtain a scaled point cloud map.

9. An electronic device, comprising a memory and a processor, the memory storing a computer program that can run on the processor, characterized in that, When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images