Map construction method, storage medium and electronic device

Abstract

The embodiment of the invention provides a map construction method, a storage medium and an electronic device. The method comprises the steps that N collected pictures and the laser radar prior pose of each collected picture in the N collected pictures are acquired, the collected pictures are pictures collected through a camera, the laser radar prior poses are poses collected through a laser radar having a connection relation with the camera, and N is a positive integer larger than or equal to 2; determining a group of three-dimensional coordinate points in a camera coordinate system corresponding to the camera according to the N collected pictures; and constructing a sparse map of a laser radar coordinate system corresponding to the laser radar according to the group of three-dimensional coordinate points and the laser radar priori poses of the N collected pictures. According to the method, the problem of scale uncertainty of a sparse map constructed by using a pure monocular vision mode in the AR field at present is solved, and the mapping precision and robustness are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision positioning, and in particular, to a map construction method, a storage medium, and an electronic device. Background Art

[0002] Currently, in the field of Augmented Reality (AR), a sparse map is mostly established by means of visual positioning, based on Structure from Motion (SFM) or Simultaneous Localization and Mapping (SLAM), and then the matching recognition of pictures is retrieved through a vocabulary tree, and the pose of the current device is determined by a pose estimation method to achieve the purpose of recognition and positioning.

[0003] However, the sparse map constructed by the above method has a scale uncertainty problem and no real scale information: in monocular vision, the scale problem refers to the situation where it is impossible to accurately measure the true size or distance of an object in the scene due to the use of only a single camera. Due to the projection nature of the camera, the depth information of the points is lost during the projection process, so a single camera cannot directly provide the absolute scale information of the objects in the scene and has scale uncertainty.

[0004] In view of the problem that the sparse map constructed by using a pure monocular vision method in the current AR field has scale uncertainty, no effective solution has been proposed yet. Summary of the Invention

[0005] The embodiments of the present application provide a map construction method, a storage medium, and an electronic device to at least solve the problem that the sparse map constructed by using a pure monocular vision method in the current AR field has scale uncertainty.

[0006] According to an embodiment of the present application, a map construction method is provided, including: obtaining N captured pictures and the prior pose of a lidar for each of the N captured pictures, where the captured pictures are pictures captured by a camera, the prior pose of the lidar is a pose captured by a lidar having a connection relationship with the camera, and N is a positive integer greater than or equal to 2; determining a set of three-dimensional coordinate points in the camera coordinate system corresponding to the camera according to the N captured pictures; and constructing a sparse map in the lidar coordinate system corresponding to the lidar according to the set of three-dimensional coordinate points and the prior pose of the lidar of the N captured pictures.

[0007] According to another embodiment of the present application, a map construction device is provided, including: an acquisition module, configured to acquire N acquisition pictures and the prior LiDAR pose of each acquisition picture among the N acquisition pictures, where the acquisition pictures are pictures acquired by a camera, and the prior LiDAR pose is a pose acquired by a LiDAR having a connection relationship with the camera, and N is a positive integer greater than or equal to 2; a mapping module, configured to determine a set of three-dimensional coordinate points in the camera coordinate system corresponding to the camera according to the N acquisition pictures, and construct a sparse map in the LiDAR coordinate system corresponding to the LiDAR according to the set of three-dimensional coordinate points and the prior LiDAR poses of the N acquisition pictures.

[0008] According to yet another embodiment of the present application, a computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium, where the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0009] According to yet another embodiment of the present application, an electronic device is further provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0010] Through the present application, in one of the above technical solutions, N acquisition pictures and the prior LiDAR pose of each acquisition picture among the N acquisition pictures are acquired, where the acquisition pictures are pictures acquired by a camera, and the prior LiDAR pose is a pose acquired by a LiDAR having a connection relationship with the camera; a set of three-dimensional coordinate points in the camera coordinate system corresponding to the camera is determined according to the N acquisition pictures; and a sparse map in the LiDAR coordinate system corresponding to the LiDAR is constructed according to the set of three-dimensional coordinate points and the prior LiDAR poses of the N acquisition pictures. Since the sparse map is constructed by jointly using the acquisition pictures and the prior LiDAR poses of the acquisition pictures, the problem of scale uncertainty in the sparse map constructed by using the pure monocular vision method in the current AR field is solved, and the mapping accuracy and robustness are improved. In addition, jointly constructing the sparse map by using the acquisition pictures and the prior LiDAR poses of the acquisition pictures can also effectively avoid scale drift and error accumulation in pure vision mapping in large scenes. Description of the Drawings

[0011] Figure 1 is a hardware structure block diagram of a computer terminal of a map construction method according to an embodiment of the present application

[0012] Figure 2 is a schematic structural diagram of a hardware device according to an embodiment of the present application;

[0013] Figure 3 It is a schematic flowchart of a map construction method according to an embodiment of the present application;

[0014] Figure 4 It is a schematic flowchart of data collection by a data collection system according to an embodiment of the present application;

[0015] Figure 5 It is a schematic flowchart of controlling the timestamp alignment of captured images and lidar according to an embodiment of the present application;

[0016] Figure 6 It is a schematic flowchart of another map construction method according to an embodiment of the present application;

[0017] Figure 7 It is a schematic flowchart of BA optimization incorporating lidar pose prior according to an embodiment of the present application;

[0018] Figure 8 It is a schematic flowchart of Sim3 transformation based on the RANSAC algorithm framework according to an embodiment of the present application;

[0019] Figure 9 It is a schematic overall flowchart of a map construction method according to an embodiment of the present application;

[0020] Figure 10 It is a schematic diagram of a sparse map of a cultural and tourism scene according to an embodiment of the present application;

[0021] Figure 11 It is a schematic diagram of a sparse map of a building scene according to an embodiment of the present application;

[0022] Figure 12 It is a structural block diagram of a map construction device according to an embodiment of the present application. Detailed implementation manners

[0023] In the following, embodiments of the present application will be described in detail with reference to the drawings and in conjunction with the embodiments.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence.

[0025] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 It is a hardware structural block diagram of a computer terminal of a map construction method according to an embodiment of the present application. As Figure 1 shown, the computer terminal may include one or more ( Figure 1Only one processor 102 is shown (the processor 102 may include, but is not limited to, a processing device such as a microcontroller unit (MCU) or a field programmable gate array (FPGA)), and a memory 104 for storing data. Among them, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 The structure shown is only schematic and does not limit the structure of the computer terminal. For example, the computer terminal may further include more or fewer components than Figure 1 shown in, or have a different configuration from Figure 1 shown.

[0026] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the map construction method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implements the above method. The memory 104 may include a high-speed random access memory and may further include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the computer terminal. In one instance, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0028] It should be noted that to implement the map construction solution of the present application, hardware device support is required. Specifically, Figure 2 A schematic diagram of a hardware device is shown. Specifically:

[0029] The hardware device consists of three parts:

[0030] (1) A terminal equipped with a Robot Operating System (ROS) (including but not limited to, for example, a laptop with the Ubuntu operating system): used to control the startup and termination of the entire data acquisition system and can observe the data acquisition situation in real time;

[0031] (2) A lidar: the livox-avia lidar, used to obtain the prior pose of the lidar. It should be noted that the prior pose of the lidar refers to the position and orientation of the lidar in the real world. The position can be represented by a three-dimensional translation vector, and the orientation can be represented by a rotation matrix (for example, a rotation matrix of dimension 3*3).

[0032] It should be noted that the camera pose in the following text refers to the position and orientation of the camera in the real world. The position can be represented by a three-dimensional translation vector, and the orientation can be represented by a rotation matrix (for example, a rotation matrix of dimension 3*3).

[0033] (3) A camera: supports running under the ROS system, used to capture pictures and obtain picture data.

[0034] It should be noted that for better understanding, the application environment of this application is specifically described as follows: This application requires supporting lidar and camera devices and a laptop installed with corresponding drivers to collect raw data, import the data into the cloud, use self-developed algorithms for SFM mapping. If the obtained sparse map file needs to be used for recognition and positioning, the map file needs to be imported into the corresponding cloud recognition library and corresponding application recognition and positioning are adopted.

[0035] In this embodiment, a map construction method is provided. Figure 3 It is a schematic flowchart of a map construction method according to an embodiment of this application, as Figure 3 shown. This process includes the following steps S302 - S306:

[0036] Step S302: Obtain N captured pictures and the prior pose of the lidar for each of the N captured pictures. Among them, the captured pictures are pictures captured by the camera, and the prior pose of the lidar is the pose captured by the lidar having a connection relationship with the camera. N is a positive integer greater than or equal to 2;

[0037] As an optional example, to ensure the mapping effect, N is a positive integer greater than or equal to 5.

[0038] As an optional example, N captured pictures and the prior pose of the lidar for each of the N captured pictures can be obtained through the data acquisition system. Specifically, the following combines Figure 4Specific description of the data acquisition system:

[0039] It should be noted that the top of the lidar and the camera base can be fixedly connected by means of components to form a stable rigid body. ROS, OpenCV, and the corresponding device drivers (realsensedriver, livox_ros_driver) are pre-installed on the laptop of the data acquisition system. After connecting the devices, start the camera and lidar drivers and collect data. During the process of collecting lidar-inertial + visual data, the lidar-inertial SLAM runs in real time. The data acquisition system will subscribe to 3 ROS topics shown in Table 1 in real time during operation, record the lidar pose and timestamp in real time, and at the same time match the image corresponding to the timestamp from the received sensor_msgs / Image message, obtain the prior pose of the lidar for the corresponding image, and output it to the image txt configuration file, so as to realize the real-time acquisition of image data containing the prior pose of the lidar.

[0040] Table 1: ROS topics subscribed by the data acquisition system

[0041] Data Topic Message Image / camera / color / image_raw sensor_msgs / Image IMU / livox / imu sensor_msgs / Imu LiDAR / livox / LiDAR livox_ros_driver / CustomMsg

[0042] In ROS, a node corresponds to a process that executes a computing task. A message is a data structure that contains field types. ROS messages support some standard data types (such as integer type, floating-point type, boolean type, etc.). In addition, these standard data types can be used to construct one's own message types. Each message in ROS is transmitted using a specific bus called a "topic". When a node sends a message through a topic, it is said that the node is publishing the topic; when a node receives a message through a topic, it is said that the node is subscribing to the topic. rosbag is mainly used to record, playback, and analyze the data in ROS topics. It can record the data in the specified topic into a data packet with the suffix ".bag", which is convenient for offline analysis and processing of the data therein.

[0043] It should be noted that the functions implemented by the data acquisition system include: outputting the pose file of the lidar; subscribing to the / camera / color / image_raw topic in real time and exporting it as an image in real time; the function of generating an image txt configuration file in real time, where the camera pose and internal parameters corresponding to the timestamp of the image are included in the txt file; supporting frame extraction of images at different frequencies in real time (which can be set in the configuration file); supporting recording rosbag while generating images in real time.

[0044] As an alternative example, when performing monocular SFM based on LiDAR pose priors, it is necessary to clarify the LiDAR pose corresponding to each image. Therefore, it is required that the timestamps of the LiDAR (Light Detection and Ranging) point cloud and the images must be aligned. The method adopted in this application is as Figure 5 shown. By using ROS commands to record the image and LiDAR topics simultaneously, the starting recording timestamps of the two are ensured to be the same. When processing each frame of LiDAR point cloud data, the timestamp corresponding to the LiDAR pose is obtained. Traverse the current image message queue. If the image timestamp and the LiDAR timestamp are inconsistent, directly delete the element at the head of the queue, that is, the element at the front position in the queue data structure. If the image timestamp and the LiDAR timestamp are consistent (the difference is the smallest and less than the threshold), export the image and the corresponding LiDAR prior pose, and jump out of the loop.

[0045] Step S304: Determine a set of three-dimensional coordinate points in the camera coordinate system corresponding to the camera according to the N captured images;

[0046] In an exemplary embodiment, the above step S304 includes: determining two captured images with Scale-Invariant Feature Transform (SIFT) feature point matching relationships from the N captured images; triangulating the two captured images to obtain the set of three-dimensional coordinate points in the camera coordinate system.

[0047] As an alternative example, before determining two captured images with SIFT feature point matching relationships from the N captured images, it is necessary to extract the SIFT feature points of each of the N captured images, and perform pairwise matching on the N captured images through the Fast Library for Approximate Nearest Neighbors (FLANN) to obtain the matching information of the N captured images.

[0048] As an alternative example, two captured images with SIFT feature point matching relationships can be randomly determined from the N captured images, or two captured images with the strongest SIFT feature point matching relationships can be determined from the N captured images.

[0049] In an exemplary embodiment, the two acquired images can be triangulated to obtain the set of three-dimensional coordinate points in the camera coordinate system in the following manner: determining a first image matching pair of the two acquired images, and determining a first fundamental matrix according to the camera poses and camera intrinsics of the two acquired images, where the camera intrinsics are the intrinsics of the camera that captured the two acquired images; using the first fundamental matrix, removing the mismatched point pairs in the first image matching pair through epipolar constraint to obtain an updated first image matching pair; projecting the pixel points in the updated first image matching pair into three-dimensional space to obtain the set of three-dimensional coordinate points.

[0050] Step S306: Construct a sparse map of the lidar in the lidar coordinate system corresponding to the lidar according to the set of three-dimensional coordinate points and the prior lidar poses of the N acquired images.

[0051] Through the above steps, a sparse map is jointly constructed using the acquired images and the prior lidar poses of the acquired images, thereby avoiding the scale uncertainty of pure monocular vision mapping and the scale drift and error accumulation of pure vision mapping in large scenes, and thus solving the problem of scale uncertainty existing in the sparse map constructed by using the pure monocular vision method in the current AR field, and improving the mapping accuracy and robustness. In addition, jointly constructing a sparse map using the acquired images and the prior lidar poses of the acquired images can also effectively avoid the scale drift and error accumulation of pure vision mapping in large scenes.

[0052] As an optional example, the above steps S302 - S306 can be executed by a map construction system, where the map construction system consists of two parts: a data acquisition and preprocessing module and a lidar and monocular fusion mapping module. The data acquisition and preprocessing module is responsible for obtaining the acquired images and their corresponding prior lidar poses, and the lidar and monocular fusion mapping module fuses and maps the above data to obtain information such as the sparse map and the feature points in the sparse map. The mapping result can be used for AR recognition and positioning. The specific flowchart is as Figure 6 shown.

[0053] In an exemplary embodiment, the above step S306 includes:

[0054] Step S11: Randomly determine a target acquired image from the M acquired images, where the M acquired images are the other acquired images in the N acquired images except the two acquired images;

[0055] Step S12: Update the set of three-dimensional coordinate points according to the target acquired image, and determine whether the updated set of three-dimensional coordinate points satisfies the optimization condition;

[0056] In an exemplary embodiment, updating the set of three-dimensional coordinate points according to the target acquisition image can be achieved in the following manner: determining a reference image from multiple acquisition images corresponding to the set of three-dimensional coordinate points, where the reference image is an acquisition image having a matching relationship with the target acquisition image; triangulating the target acquisition image and the reference image to obtain a plurality of three-dimensional coordinate points, and adding the plurality of three-dimensional coordinate points to the set of three-dimensional coordinate points.

[0057] In an exemplary embodiment, determining whether the updated set of three-dimensional coordinate points meets the optimization condition includes: determining a first quantity of the three-dimensional coordinate points in the set of three-dimensional coordinate points before updating and a second quantity of the acquisition images corresponding to the set of three-dimensional coordinate points before updating; and determining a third quantity of the three-dimensional coordinate points in the updated set of three-dimensional coordinate points and a fourth quantity of the acquisition images corresponding to the updated set of three-dimensional coordinate points; determining that the optimization condition is met when the ratio of the first difference to the first quantity is greater than or equal to a first threshold and the ratio of the second difference to the second quantity is greater than or equal to a second threshold, where the first difference is the difference between the third quantity and the first quantity, and the second difference is the difference between the fourth quantity and the second quantity; determining that the optimization condition is not met when the ratio of the first difference to the first quantity is less than the first threshold or the ratio of the second difference to the second quantity is less than the second threshold.

[0058] As an alternative example, both the first threshold and the second threshold may be 20%, that is, if the number of acquisition images corresponding to a set of three-dimensional coordinate points increases by more than 20% and the number of corresponding three-dimensional coordinate points increases by more than 20%, it is determined that the optimization condition is met.

[0059] Step S13: When the set of three-dimensional coordinate points meets the optimization condition, performing bundle adjustment (BA) optimization on the set of three-dimensional coordinate points using the first lidar prior pose set as a constraint to obtain the sparse map, where the first lidar prior pose set includes: the lidar prior poses of each acquisition image in multiple acquisition images corresponding to the set of three-dimensional coordinate points;

[0060] An iterative BA optimization algorithm incorporating lidar pose prior proposed in this application constrains a set of three-dimensional coordinate points through the prior information of the lidar pose during the iterative BA process to prevent error accumulation and scale drift during large-scale SFM mapping.

[0061] In an exemplary embodiment, the above step S13 can be implemented through the following steps S131 - S134:

[0062] Step S131: Use the target similarity transformation matrix to transform the set of three-dimensional coordinate points and the camera poses of the multiple captured images corresponding to the set of three-dimensional coordinate points from the camera coordinate system to the lidar coordinate system, where the target similarity transformation matrix is used to transform the camera poses of the N captured images from the camera coordinate system to the lidar coordinate system and to transform a set of three-dimensional coordinate points in the camera coordinate system to a set of three-dimensional coordinate points in the lidar coordinate system;

[0063] Step S132: Construct a non-linear optimization model based on the target position error and the reprojection error corresponding to the set of three-dimensional coordinate points, and optimize the set of three-dimensional coordinate points and the camera poses of the multiple captured images corresponding to the set of three-dimensional coordinate points according to the non-linear optimization model, where the target position error is the position error between the camera poses of the multiple captured images corresponding to the set of three-dimensional coordinate points and the lidar prior poses of the multiple captured images;

[0064] Step S133: Remove the three-dimensional coordinate points in the set of three-dimensional coordinate points whose reprojection error is greater than the third threshold;

[0065] In an exemplary embodiment, the reprojection error of the i-th three-dimensional coordinate point in the set of three-dimensional coordinate points can be determined in the following manner to determine the reprojection error of each three-dimensional coordinate point in the set of three-dimensional coordinate points: Obtain multiple observations of the i-th three-dimensional coordinate point, and calculate the reprojection error corresponding to each observation according to the camera pose of the captured image corresponding to each observation in the multiple observations, obtaining multiple reprojection errors; Determine the average value of the multiple reprojection errors as the reprojection error of the i-th three-dimensional coordinate point.

[0066] It should be noted that assuming that the i-th three-dimensional coordinate point corresponds to 3 captured images among the N captured images, then the i-th three-dimensional coordinate point has 3 observations (one observation corresponding to each of the 3 captured images of the i-th three-dimensional coordinate point on each captured image).

[0067] Step S134: Re-triangulate and update the set of three-dimensional coordinate points according to the optimized camera poses of the multiple captured images to obtain the sparse map, where the sparse map includes the set of three-dimensional coordinate points in the lidar coordinate system.

[0068] In an exemplary embodiment, the above step S134 includes: traversing the optimized multiple captured images, and performing the following operations on each captured image, where the current captured image is the currently traversed captured image: determining a reference captured image having a matching relationship with the current captured image from the multiple captured images; determining a second image matching pair of the current captured image and the reference captured image, and determining a second fundamental matrix according to the camera poses of the current captured image and the reference captured image; using the second fundamental matrix to eliminate mismatched point pairs in the second image matching pair through epipolar constraint to obtain an updated second image matching pair; projecting the target pixel points in the second image matching pair into three-dimensional space, and adding the projected three-dimensional coordinate points to the set of three-dimensional coordinate points, where the target pixel points are the pixel points in the second image matching pair that have no corresponding relationship in the set of three-dimensional coordinate points.

[0069] As an alternative example, determining a second fundamental matrix according to the camera poses of the current captured image and the reference captured image includes: determining a second fundamental matrix according to the camera poses and camera intrinsics of the current captured image and the reference captured image.

[0070] As an alternative example, after adding the projected three-dimensional coordinate points to the set of three-dimensional coordinate points, the method further includes: eliminating the three-dimensional coordinate points in the set of three-dimensional coordinate points that do not meet the preset conditions, where the preset conditions include: the three-dimensional coordinate points are bounded, the three-dimensional coordinate points are all in front of the cameras that captured the first captured image and the second captured image, the reprojection errors of the three-dimensional coordinate points corresponding to the first captured image and the second captured image are both less than a fourth threshold, the three-dimensional coordinate points, the first optical center, and the second optical center cannot form a straight line, and the absolute values of the cosines of the three angles of the triangle formed by the three-dimensional coordinate points and the first optical center and the second optical center are less than a fifth threshold; the first captured image and the second captured image are two captured images in the multiple captured images corresponding to the three-dimensional coordinate points, the first optical center is the optical center of the camera when capturing the first captured image, and the second optical center is the optical center of the camera when capturing the second captured image.

[0071] It should be noted that the three-dimensional coordinate points being bounded means that the coordinates of the three-dimensional coordinate points are within a preset range.

[0072] For better understanding, the following Figure 7 specifically describes the above BA optimization, where the execution process of each iteration of the BA optimization process is as follows:

[0073] (1) Adding priors: Transforming the camera poses and map points (i.e., three-dimensional coordinate points) in SFM to the LiDAR coordinate system through the Sim3 transformation under the RANSAC framework;

[0074] (2) BA optimization incorporating lidar pose prior: A non-linear optimization model is constructed based on the position error between the lidar prior pose and the camera pose to be optimized, and added to the final global BA optimization process. The camera pose and map point coordinates are optimized by minimizing the reprojection error and position error.

[0075] (3) Map point filtering: Calculate the reprojection error of each map point. If the reprojection error of a map point exceeds the set third threshold, the point is considered an outlier, and the corresponding map point and all its related observations are removed.

[0076] The specific process is as follows: Traverse all map points in the existing map, and perform the following operations for each map point:

[0077] a) Traverse all observations corresponding to the map point; b) For each observation, calculate the reprojection error value according to the optimized key frame pose, camera internal parameters, distortion parameters, and two-dimensional pixel point coordinates according to the pinhole camera model; c) Calculate the average value of the reprojection errors of all observations corresponding to the point by referring to the method in the previous step; d) Judge whether the average value of the reprojection error of the point exceeds the set third threshold. If it exceeds, the point is considered an abnormal point and is removed from the map, otherwise the point is retained.

[0078] (4) Repro-triangulation: New map points that meet the requirements are re-triangulated according to the new camera pose obtained by BA optimization. The specific process is to traverse all key frames (i.e., captured pictures) that have been added to the map. For each key frame, perform the following operations:

[0079] a) Find other key frames in the map that have a matching relationship with the key frame as reference key frames, and the two frames form an image matching pair; b) Solve the pose transformation between the two frames according to the key frame pose and the reference frame pose, and solve the fundamental matrix; c) Traverse each matching point pair in the image matching pair, and remove false matches through epipolar constraint. If the key point has been triangulated to recover the map point before, skip it; d) Triangulate the matching point pair from the previous step to obtain a new map point, i.e., a 3D point; e) Traverse and check whether all triangulated map points meet the following four conditions: bounded, i.e., the point is not an infinite point; the point is in front of both cameras; the reprojection error of the point in both cameras is less than the threshold; the triangulation relationship between the point and the optical centers of the two cameras needs to meet the condition that the three points cannot form a straight line and the absolute value of the cosine of the three angles is less than the set threshold; f) Add the new map points that meet the inspection conditions to the map, add map points to the current key frame at the same time, add observations to each map point, and calculate the best descriptor for each map point; g) Add the new key frame to the key frame sequence of the map.

[0080] Step S14: Loop through the above steps S11 - S13 until the multiple captured images corresponding to the set of three - dimensional coordinate points are the N captured images, where the target captured image determined each time is different from the previously determined target captured image.

[0081] That is to say, in this embodiment, after obtaining a set of three - dimensional coordinate points from two captured images among the N captured images, for the remaining captured images among the N captured images, they need to be added to the sparse map in sequence. During the addition process, search for all captured images that have a matching relationship with this captured image and have already been added to the map, triangulate to obtain new map points based on the matching information, and determine whether the optimization condition is met. If it is met, use the information of all captured images and map points corresponding to the existing sparse map to perform iterative BA optimization incorporating the lidar pose prior.

[0082] In an exemplary embodiment, after obtaining the sparse map, the method further includes: using the first lidar prior pose set as a constraint to perform BA optimization on the set of three - dimensional coordinate points for a target number of times to obtain the sparse map.

[0083] As an optional example, the above - mentioned target number of times is 3 times. It should be noted that due to the influence of outliers, there is still room for reducing the errors of map points and poses in a single BA. Therefore, the iterative BA method is used to improve the accuracy of the final map points. Iterative BA is called when the incremental reconstruction meets the condition of "the number of key frames increases by more than 20% and the number of map points increases by more than 20%", and it is only iterated once; in the final global BA, it is iterated 3 times.

[0084] In an exemplary embodiment, when using the lidar prior pose as a prior constraint for the camera pose to be optimized, only the position prior is considered. Let be the three - dimensional coordinates of the camera in the world coordinate system after undergoing a similarity transformation, be the three - dimensional coordinates of the corresponding lidar in the world coordinate system. Then the observation equation in the world coordinate system should be:

[0085]

[0086] Since the camera pose to be optimized is where are the rotation matrix and translation vector from the world coordinate system to the camera coordinate system respectively. Therefore, to convert the coordinates in the above - mentioned observation equation to coordinates, the transformed observation equation should be:

[0087]

[0088] According to the above formula, a cost function is constructed based on the position error between the prior pose of the lidar and the camera pose to be optimized, and it is incorporated into the BA optimization as a unary edge of the General Graph Optimization (abbreviated as g2o, a third-party library for BA optimization).

[0089] In an exemplary embodiment, before performing bundle adjustment (BA) optimization on the set of three-dimensional coordinate points using the first set of lidar prior poses as constraints, the method further includes: determining the camera pose of each of the N captured images to obtain N camera poses; calculating a target similarity transformation matrix based on the similarity transformation of the Random Sample Consensus (RANSAC) algorithm according to the N camera poses and the N lidar prior poses corresponding to the N captured images; wherein, the target similarity transformation matrix is used to transform the N camera poses from the camera coordinate system to the lidar coordinate system, and to transform a set of three-dimensional coordinate points in the camera coordinate system to a set of three-dimensional coordinate points in the lidar coordinate system.

[0090] In an exemplary embodiment, calculating a target similarity transformation matrix based on the similarity transformation of the Random Sample Consensus (RANSAC) algorithm according to the N camera poses and the N lidar prior poses corresponding to the N captured images includes the following steps S21 - S22:

[0091] Step S21: Initialize the number of iterations, the minimum required number of iterations, the maximum allowed number of iterations, the optimal number of inliers, and the optimal average error, wherein the initialized number of iterations is 0;

[0092] Step S22: Loop through the following steps S221 - S224 to exit the loop when the number of iterations is equal to the maximum allowed number of iterations, or when the number of iterations is greater than the maximum number of iterations and greater than the minimum required number of iterations, to obtain a set of similarity transformation matrices, and determine the reference similarity transformation matrix that meets the preset conditions in the set of similarity transformation matrices as the target similarity transformation matrix, wherein the preset conditions include: the largest number of inliers and the smallest average error:

[0093] Step S221: Randomly select a preset number of camera poses from the N camera poses, and determine a reference similarity transformation matrix according to the preset number of camera poses and the second set of lidar prior poses, wherein the second set of lidar prior poses includes the lidar prior poses corresponding to the preset number of camera poses;

[0094] Step S222: Determine the relevant information of the reference similarity transformation matrix according to the reference similarity transformation matrix, and record the reference similarity transformation matrix and the relevant information of the reference similarity transformation matrix in the similarity transformation matrix set, where the relevant information includes: the number of inliers among the N camera poses and the average error of all inliers;

[0095] Step S223: Calculate the maximum number of iterations of the RANSAC algorithm according to the relevant information of the reference similarity transformation matrix;

[0096] Step S224: In the case where the number of iterations is less than or equal to the maximum number of iterations or less than or equal to the minimum required number of iterations, increment the number of iterations by one.

[0097] In an exemplary embodiment, calculating the maximum number of iterations of the RANSAC algorithm according to the relevant information of the reference similarity transformation matrix includes: in the case where the number of inliers is greater than or equal to the optimal number of inliers and the average error is less than the optimal average error, updating the optimal number of inliers to the number of inliers; calculating the maximum number of iterations of the RANSAC algorithm according to the optimal number of inliers; in the case where the number of inliers is less than the optimal number of inliers or the average error is greater than or equal to the optimal average error, calculating the maximum number of iterations of the RANSAC algorithm according to the optimal number of inliers.

[0098] In an exemplary embodiment, after updating the optimal number of inliers to the number of inliers, the method further includes: before calculating the maximum number of iterations of the RANSAC algorithm according to the optimal number of inliers, updating the optimal average error to the average error, re-determining the reference similarity transformation matrix using all inliers among the N camera poses and the lidar prior poses corresponding to all inliers; re-determining the relevant information of the reference similarity transformation matrix according to the reference similarity transformation matrix; recording the reference similarity transformation matrix and the relevant information of the reference similarity transformation matrix in the similarity transformation matrix set again; in the case where the number of inliers is greater than or equal to the optimal number of inliers and the average error is less than the optimal average error, updating the optimal number of inliers to the number of inliers and updating the optimal average error to the average error.

[0099] It should be noted that for a better understanding of the above determination of the target similarity transformation matrix, the following is a specific description:

[0100] When obtaining the visual acquisition images used for mapping and the corresponding LiDAR prior pose, it is necessary to take the camera coordinate system and the LiDAR coordinate system together to solve the similarity transformation matrix T between the two. The defect of the traditional Sim3 transformation based purely on the least squares algorithm is that it is sensitive to outliers, and when there are gross errors in the pose, it has a greater impact on the calculation results. The present application solves the Sim3 transformation between the LiDAR prior pose and the SFM camera pose based on the RANSAC algorithm framework, which can overcome the defects of the traditional Sim3 transformation based purely on the least squares algorithm, which is sensitive to outliers and easily causes large deviations in the calculation results when there are gross errors in the input pose. It can eliminate outliers in the LiDAR prior pose or camera pose to improve the robustness of the final similarity transformation matrix and further improve the accuracy of mapping and recognition. Assume that in the minimum sample subset of the RANSAC algorithm, the corresponding 3D point coordinates in the original coordinate system and the target coordinate system are x and y, respectively. i ,y i (i=1,2,...,n), the Umeyama algorithm is used to solve the Sim3 transformation between two types of poses in the smallest subset. The algorithm is based on least squares to solve the optimal similarity transformation matrix T:

[0101] Where R, t, and c are rotation matrices, translation vectors, and scale parameters, respectively. The cost function is as follows:

[0102]

[0103] The Sim3 transformation based on the RANSAC algorithm continuously selects the smallest subset randomly from all sample data through iteration, calculates the Sim3 transformation between the LiDAR and camera coordinate systems according to the above Umeyama algorithm, selects a set of solutions with the largest number of internal points and the smallest average error as the current optimal solution, and obtains the target similarity transformation matrix until the number of iterations exceeds the theoretical maximum number of iterations allowed. The specific flow chart is as follows Figure 8 As shown, Figure 8 The minimum sample subset in is the above-mentioned preset number of camera poses and the second lidar prior pose set.

[0104] For better understanding, Figure 9 The overall flow chart of the map construction method of this application is shown as follows: Figure 9 As shown in the figure, the real-time data acquisition system is first used to obtain image data with lidar pose prior. After feature extraction, matching and initialization, the lidar pose prior and camera pose are unified into the same coordinate system through Sim3 transformation under the RANSAC framework during the SFM incremental mapping process. The lidar pose prior is taken as a constraint and incorporated into the iterative BA optimization to obtain the best estimate of the map points and camera pose. Finally, the obtained sparse map is used for recognition and positioning.

[0105] Obviously, the embodiments described above are only a part of the embodiments of this application, rather than all of them. To better understand the above method, the following will describe the above process in combination with embodiments, but it is not used to limit the technical solutions of the embodiments of this application. Specifically:

[0106] Embodiment 1: Mapping, recognition and positioning in the cultural and tourism scenario

[0107] Step1: Connect the devices. The lidar model is livox-avia, and the camera model is intel realsenseD435i. Run the one-key launch file to achieve the functions of starting roscore and starting the device driver (the driver needs to be started after connecting the device to ensure the normal operation of the device): roslaunch run_driver.launch;

[0108] Step2: Create a new tab in the command-line terminal and run the laser SLAM developed based on FAST-LIO. The parameters during operation are set in the mapping_avia.launch file. The custom parameters include the data saving path, camera frame rate, image sampling rate, camera internal parameters, etc.;

[0109] After startup, the following content is displayed in real time on the command-line interface: the mapping information of the cultural and tourism scenario by the laser inertial SLAM; the timestamps of the current frame LiDAR point cloud and the image message queue; the image with the smallest time difference from the current frame LiDAR point cloud timestamp in the message queue and the exported information of its lidar prior pose;

[0110] Step3: Take the pictures of the cultural and tourism scenario obtained in Step2 and their lidar prior poses as inputs, and perform monocular SFM mapping with lidar pose prior constraints according to the above-provided laser and monocular fusion mapping module. All processes of this module have been integrated into a complete algorithm package. When the corresponding data is input into the algorithm package, it will automatically run and solve to obtain the sparse map of the cultural and tourism scenario and the feature points and descriptor information corresponding to the map points. The specific process is as follows:

[0111] (1) Feature extraction, matching and initialization

[0112] Extract the SIFT feature points of all pictures of the cultural and tourism scenario, perform pairwise matching on all pictures through FLANN to obtain the matching information of all pictures, select two appropriate pictures and triangulate them according to their matching feature points to obtain several sparse 3D points, and realize SFM initialization.

[0113] (2) Incremental reconstruction

[0114] For the remaining images after initialization, add them to the map one by one. During the addition process, search for all the images that have a matching relationship with the current image and have already been added to the map, and triangulate new map points based on the matching information. Determine whether the BA condition is satisfied. If it is satisfied, perform iterative BA optimization incorporating the lidar pose prior using all the existing frame images and map point information.

[0115] (3) Global iterative BA optimization

[0116] After incremental mapping is completed, perform another iterative BA optimization incorporating the lidar pose prior on all the image and map point information to obtain the final sparse map and the information of the map points corresponding to the image feature points.

[0117] Step4: Perform relocalization recognition based on the reconstructed sparse map of the cultural and tourism scene: Obtain the matching information of the input recognition image based on the previous sparse map and its feature point information, and estimate the pose of the current recognition image through an algorithm for solving the camera pose based on 3D-2D point pairs (e.g., Perspective-n-Point, abbreviated as PNP algorithm) to achieve relocalization. After successful recognition, model choreography can be added to achieve the effect of combining virtual and real.

[0118] As an optional example, Figure 10 Schematically shows a sparse map of a cultural and tourism scene.

[0119] Example 2: Mapping and recognition localization of building scenes

[0120] Step1: Connect the devices. The lidar model is livox-avia, and the camera model is intel realsenseD435i. Run the one-key launch file to achieve the functions of starting roscore and starting the device driver (the driver needs to be started after connecting the device to ensure the normal operation of the device): roslaunch run_driver.launch;

[0121] Step2: Create a new tab in the command-line terminal and run the laser SLAM developed based on FAST-LIO. The parameters during operation are set in the mapping_avia.launch file. The custom parameters include the data saving path, camera frame rate, image sampling rate, camera internal parameters, etc.;

[0122] After startup, the following content is displayed in real time on the command-line interface: the mapping information of the building scene by the laser inertial SLAM; the time stamps of the current frame LiDAR point cloud and the image message queue; the image with the smallest time stamp difference from the current frame LiDAR point cloud in the message queue and its prior pose export information;

[0123] Step 3: Take the images of the building scene obtained in Step 2 and their prior poses as inputs, and perform monocular SFM mapping incorporating the prior pose of the lidar according to the lidar and monocular fusion mapping module provided in 7.3. All processes of this module have been integrated into a complete algorithm package. When the corresponding data is input into the algorithm package, it will automatically run and solve to obtain the sparse map of the building scene, as well as the feature points and descriptor information corresponding to the map points. The specific process is as follows:

[0124] (1) Feature extraction, matching, and initialization

[0125] Extract the SIFT feature points of all images of the building scene, perform pairwise matching on all images through FLANN to obtain the matching information of all images, select two appropriate images, and triangulate them based on their matching feature points to obtain several sparse 3D points, realizing the initialization of SFM.

[0126] (2) Incremental reconstruction

[0127] For the remaining images after initialization, add them to the map one by one. During the addition process, search for all images that have a matching relationship with this image and have already been added to the map, and triangulate them based on the matching information to obtain new map points. Judge whether the BA condition is satisfied. If it is satisfied, perform iterative BA optimization incorporating the prior pose of the lidar using all the existing frame images and map point information.

[0128] (3) Global iterative BA optimization

[0129] After the incremental mapping is completed, perform another iterative BA optimization incorporating the prior pose of the lidar on all the image and map point information to obtain the final sparse map and the information of the image feature points corresponding to the map points.

[0130] Step 4: Perform relocalization recognition based on the reconstructed sparse map of the building scene: Obtain the matching information of the input recognition image according to the previous sparse map and its feature point information, and estimate the pose of the current recognition image through an algorithm for solving the camera pose based on 3D-2D point pairs (for example: Perspective-n-Point, abbreviated as PNP algorithm) to achieve relocalization.

[0131] As an optional example, Figure 11 A sparse map of a building scene is shown.

[0132] It should be noted that an iterative BA optimization algorithm incorporating lidar pose prior proposed in this application constrains the camera pose through the prior information of the lidar pose during the iterative BA process to prevent error accumulation and scale drift during large-scale SFM mapping. The position error of the camera relative to the lidar is minimized as the cost function. In each iteration, an outlier filtering and re-triangulation strategy is introduced to calculate the reprojection error of each map point and filter out the points whose errors exceed the set threshold. New map points that meet the requirements are re-triangulated based on the new camera pose obtained by BA optimization. At the same time, a robust kernel function is introduced to improve the mapping accuracy and robustness.

[0133] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as Read-Only Memory / Random Access Memory, ROM / RAM, magnetic disk, optical disk), and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

[0134] In this embodiment, a map construction device is also provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0135] Figure 12 is a structural block diagram of the map construction device according to an embodiment of this application. As Figure 12 shown, the device includes:

[0136] An acquisition module 122, configured to acquire N acquisition pictures and the lidar prior pose of each acquisition picture among the N acquisition pictures, where the acquisition pictures are pictures acquired by a camera, the lidar prior pose is the pose acquired by a lidar having a connection relationship with the camera, and N is a positive integer greater than or equal to 2;

[0137] A mapping module 124, configured to determine a set of three-dimensional coordinate points in the camera coordinate system corresponding to the camera according to the N captured images, and construct a sparse map in the lidar coordinate system corresponding to the lidar according to the set of three-dimensional coordinate points and the prior pose of the lidar in the N captured images.

[0138] With the above device, a sparse map is jointly constructed by using the captured images and the prior pose of the lidar in the captured images, thereby solving the problem of scale uncertainty existing in the sparse map constructed by using the pure monocular vision method in the current AR field, and improving the mapping accuracy and robustness. In addition, jointly constructing a sparse map by using the captured images and the prior pose of the lidar in the captured images can also effectively avoid scale drift and error accumulation in pure vision mapping in large scenes.

[0139] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above-mentioned modules are all located in the same processor; or, the above-mentioned various modules are respectively located in different processors in any combination form.

[0140] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0141] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drive, read-only memory (abbreviated as ROM), random access memory (abbreviated as RAM), mobile hard disk, magnetic disk or optical disc and other various media that can store computer programs.

[0142] An embodiment of the present application also provides an electronic device, including a memory and a processor, a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0143] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0144] The specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be repeated here.

[0145] Obviously, those skilled in the art should understand that the various modules or steps of the present application described above can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a sequence different from that here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0146] The foregoing is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for map construction, characterized in that, it includes: obtaining N captured images and the prior pose of a lidar for each of the N captured images, where the captured images are images captured by a camera, the prior pose of the lidar is the pose captured by a lidar having a connection relationship with the camera, and N is a positive integer greater than or equal to 2; determining a set of three-dimensional coordinate points in the camera coordinate system corresponding to the camera according to the N captured images; constructing a sparse map in the lidar coordinate system corresponding to the lidar according to the set of three-dimensional coordinate points and the prior pose of the lidar of the N captured images.

2. The method according to claim 1, characterized in that, determining a set of three-dimensional coordinate points in the camera coordinate system corresponding to the camera according to the N captured images includes: determining two captured images having a Scale-Invariant Feature Transform (SIFT) feature point matching relationship from the N captured images; triangulating the two captured images to obtain the set of three-dimensional coordinate points located in the camera coordinate system.

3. The method according to claim 2, characterized in that, constructing a sparse map in the lidar coordinate system corresponding to the lidar according to the set of three-dimensional coordinate points and the prior pose of the lidar of the N captured images includes: randomly determining a target captured image from M captured images, where the M captured images are the other captured images among the N captured images except the two captured images; updating the set of three-dimensional coordinate points according to the target captured image, and determining whether the updated set of three-dimensional coordinate points meets the optimization condition; when the set of three-dimensional coordinate points meets the optimization condition, performing Bundle Adjustment (BA) optimization on the set of three-dimensional coordinate points using a first set of prior lidar poses as constraints to obtain the sparse map, where the first set of prior lidar poses includes: the prior lidar pose of each of the multiple captured images corresponding to the set of three-dimensional coordinate points; repeatedly executing the above steps until the multiple captured images corresponding to the set of three-dimensional coordinate points are the N captured images, where the target captured image determined each time is different from the target captured image determined previously.

4. The method according to claim 3, characterized in that, determining whether the updated set of three-dimensional coordinate points meets the optimization condition includes: determining the first quantity of the three-dimensional coordinate points in the set of three-dimensional coordinate points before update and the second quantity of the captured images corresponding to the set of three-dimensional coordinate points before update; and determining the third quantity of the three-dimensional coordinate points in the updated set of three-dimensional coordinate points and the fourth quantity of the captured images corresponding to the updated set of three-dimensional coordinate points; when the ratio of the first difference to the first quantity is greater than or equal to a first threshold and the ratio of the second difference to the second quantity is greater than or equal to a second threshold, it is determined that the optimization condition is met, where the first difference is the difference between the third quantity and the first quantity, and the second difference is the difference between the fourth quantity and the second quantity.

5. The method according to claim 3, wherein, before performing bundle adjustment (BA) optimization on the set of three-dimensional coordinate points using the first lidar prior pose set as a constraint, the method further includes: determining the camera pose of each of the N captured images in the N captured images to obtain N camera poses; calculating a target similarity transformation matrix based on a similarity transformation of the random sample consensus (RANSAC) algorithm according to the N camera poses and the N lidar prior poses corresponding to the N captured images; wherein, the target similarity transformation matrix is used to transform the N camera poses from the camera coordinate system to the lidar coordinate system, and to transform a set of three-dimensional coordinate points in the camera coordinate system to a set of three-dimensional coordinate points in the lidar coordinate system.

6. The method according to claim 5, wherein, calculating a target similarity transformation matrix based on a similarity transformation of the random sample consensus (RANSAC) algorithm according to the N camera poses and the N lidar prior poses corresponding to the N captured images includes: initializing the number of iterations, the minimum required number of iterations, the maximum allowed number of iterations, the optimal number of inliers, and the optimal average error, wherein the initialized number of iterations is 0; repeatedly executing the following steps to exit the loop when the number of iterations is equal to the maximum allowed number of iterations, or when the number of iterations is greater than the maximum number of iterations and greater than the minimum required number of iterations, to obtain a set of similarity transformation matrices, and determining the reference similarity transformation matrix that satisfies a preset condition in the set of similarity transformation matrices as the target similarity transformation matrix, wherein the preset condition includes: the largest number of inliers and the smallest average error: randomly selecting a preset number of camera poses from the N camera poses, and determining a reference similarity transformation matrix according to the preset number of camera poses and a second lidar prior pose set, wherein the second lidar prior pose set includes the lidar prior poses corresponding to the preset number of camera poses; determining the relevant information of the reference similarity transformation matrix according to the reference similarity transformation matrix, and recording the reference similarity transformation matrix and the relevant information of the reference similarity transformation matrix in the set of similarity transformation matrices, wherein the relevant information includes: the number of inliers among the N camera poses and the average error of all inliers; calculating the maximum number of iterations of the RANSAC algorithm according to the relevant information of the reference similarity transformation matrix; when the number of iterations is less than or equal to the maximum number of iterations, or less than or equal to the minimum required number of iterations, incrementing the number of iterations by one.

7. The method according to claim 6, wherein, calculating the maximum number of iterations of the RANSAC algorithm according to the relevant information of the reference similarity transformation matrix includes: when the number of inliers is greater than or equal to the optimal number of inliers and the average error is less than the optimal average error, updating the optimal number of inliers to the number of inliers; calculating the maximum number of iterations of the RANSAC algorithm according to the optimal number of inliers; In the case where the number of inliers is less than the optimal number of inliers, or the average error is greater than or equal to the optimal average error, calculate the maximum number of iterations of the RANSAC algorithm according to the optimal number of inliers.

8. The method according to claim 7, wherein, after updating the optimal number of inliers to the number of inliers, the method further includes: before calculating the maximum number of iterations of the RANSAC algorithm according to the optimal number of inliers, update the optimal average error to the average error, and re-determine the reference similarity transformation matrix using all inliers in the N camera poses and the lidar prior poses corresponding to all the inliers; re-determine the relevant information of the reference similarity transformation matrix according to the reference similarity transformation matrix; record the reference similarity transformation matrix and the relevant information of the reference similarity transformation matrix in the similarity transformation matrix set again; in the case where the number of inliers is greater than or equal to the optimal number of inliers and the average error is less than the optimal average error, update the optimal number of inliers to the number of inliers and update the optimal average error to the average error.

9. The method according to claim 3, wherein, performing bundle adjustment (BA) optimization on the set of three-dimensional coordinate points using the first lidar prior pose set as a constraint to obtain a sparse map, including: transforming the set of three-dimensional coordinate points and the camera poses of multiple captured images corresponding to the set of three-dimensional coordinate points from the camera coordinate system to the lidar coordinate system using a target similarity transformation matrix, wherein the target similarity transformation matrix is used to transform the camera poses of the N captured images from the camera coordinate system to the lidar coordinate system and to transform a set of three-dimensional coordinate points in the camera coordinate system to a set of three-dimensional coordinate points in the lidar coordinate system; constructing a non-linear optimization model according to the target position error and the reprojection error corresponding to the set of three-dimensional coordinate points, and optimizing the set of three-dimensional coordinate points and the camera poses of multiple captured images corresponding to the set of three-dimensional coordinate points according to the non-linear optimization model, wherein the target position error is the position error between the camera poses of multiple captured images corresponding to the set of three-dimensional coordinate points and the lidar prior poses of the multiple captured images; removing the three-dimensional coordinate points in the set of three-dimensional coordinate points whose reprojection error is greater than a third threshold; re-triangulating and updating the set of three-dimensional coordinate points according to the optimized camera poses of the multiple captured images to obtain the sparse map, wherein the sparse map includes the set of three-dimensional coordinate points in the lidar coordinate system.

10. The method according to claim 9, wherein, re-triangulating and updating the set of three-dimensional coordinate points according to the optimized camera poses of the multiple captured images includes: traversing the optimized multiple captured images, and performing the following operations on each captured image, where the current captured image is the currently traversed captured image: determine a reference captured image having a matching relationship with the current captured image from the multiple captured images; Determine the second image matching pairs of the current captured image and the reference captured image, and determine the second fundamental matrix according to the camera poses of the current captured image and the reference captured image; Using the second fundamental matrix, eliminate the mismatched point pairs in the second image matching pairs to obtain the updated second image matching pairs; Project the target pixel points in the second image matching pairs into the three-dimensional space, and add the projected three-dimensional coordinate points to the set of three-dimensional coordinate points, where the target pixel points are the pixel points in the second image matching pairs that have no corresponding relationship in the set of three-dimensional coordinate points.

11. The method according to claim 10, wherein, After adding the projected three-dimensional coordinate points to the set of three-dimensional coordinate points, the method further includes: Eliminate the three-dimensional coordinate points in the set of three-dimensional coordinate points that do not meet the preset conditions, where the preset conditions include: the three-dimensional coordinate points are bounded, the three-dimensional coordinate points are all in front of the camera that captured the first captured image and the second captured image, the reprojection errors of the three-dimensional coordinate points corresponding to the first captured image and the second captured image are both less than the fourth threshold, the three-dimensional coordinate points, the first optical center and the second optical center cannot form a straight line, and the absolute values of the cosines of the three angles of the triangle formed by the three-dimensional coordinate points and the first optical center and the second optical center are less than the fifth threshold; the first captured image and the second captured image are the captured images corresponding to the three-dimensional coordinate points in the multiple captured images, the first optical center is the optical center when the camera captures the first captured image, and the second optical center is the optical center when the camera captures the second captured image.

12. The method according to claim 3, wherein, After obtaining the sparse map, the method further includes: Using the first lidar prior pose set as a constraint, perform bundle adjustment (BA) optimization on the set of three-dimensional coordinate points for the target number of times to obtain the sparse map.

13. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 12 are implemented.

14. An electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the computer program, the steps of the method described in any one of claims 1 to 12 are implemented.

Images