Map construction method, system, device, and computer storage medium

Abstract

The application discloses a map construction method, system, device and computer storage medium. The map construction method comprises the following steps: receiving image data and inertial data; obtaining three-dimensional coordinate points of the map based on the image data and the inertial data; generating plane data by using the three-dimensional coordinate points; and constructing the map by combining the three-dimensional coordinate points and the plane data. The map construction method of the application realizes tight coupling of image data, inertial data and plane data, effectively improves positioning of the environment through the plane data, provides accurate short-time tracking through the inertial data, and realizes real-time and efficient pose and map estimation in a weak-texture environment.

Description

Technical Field

[0001] The present application relates to the field of computer vision processing technology, and in particular to a map construction method, system, device and computer storage medium. Background Art

[0002] Visual Simultaneous Localization and Mapping (SLAM) has been extensively researched over the past 20 years and has been widely applied in emerging fields such as robotics, autonomous driving, and AR / VR (Augmented Reality). Existing visual SLAM algorithms can be categorized based on front-end tracking methods: indirect methods based on feature points and direct methods based on grayscale information. Back-end optimization methods can be further categorized into filtering-based and bundle adjustment-based approaches.

[0003] In single-objective SLAM, due to the camera's limited field of view, both feature point-based and direct methods are not robust enough in situations with intense motion, changing lighting, or sparse textures. Panoramic vision sensors, on the other hand, can capture 360° of surrounding scene information, effectively addressing this limited field of view. However, due to inherent limitations of vision sensors, tracking loss can still occur even with panoramic cameras. Summary of the Invention

[0004] The present application provides a map construction method, system, device and computer storage medium.

[0005] A technical solution adopted by this application is to provide a map construction method, which includes:

[0006] Receive image data and inertial data;

[0007] Acquiring three-dimensional coordinate points of the map based on the image data and the inertial data;

[0008] generating plane data using the three-dimensional coordinate points;

[0009] The map is constructed by combining the three-dimensional coordinate points and the plane data.

[0010] After generating plane data using the three-dimensional coordinate points, the map construction method further includes:

[0011] obtaining an inertial constraint using the inertial data;

[0012] Obtaining a reprojection error and a depth constraint using the plane data and the three-dimensional coordinate points;

[0013] generating a first plane optimization function according to the inertia constraint, the reprojection error, and the depth constraint;

[0014] The plane data is optimized using the first plane optimization function.

[0015] The step of generating a first plane optimization function according to the inertia constraint, the reprojection error, and the depth constraint includes:

[0016] Based on the plane data and the three-dimensional coordinate point, obtaining a first coordinate of an observation point associated with the plane in a first key frame, and a second coordinate of the observation point in a second key frame;

[0017] Obtaining a point-plane constraint between the observation point and the plane using the first coordinate, the second coordinate, and a preset projection equation;

[0018] A first plane optimization function is generated according to the point-plane constraint, the inertia constraint, the reprojection error, and the depth constraint.

[0019] After generating plane data using the three-dimensional coordinate points, the map construction method further includes:

[0020] Based on the image data, obtaining first camera parameters for capturing a first key frame and second camera parameters for capturing a second key frame;

[0021] Obtaining relative camera parameters of the first camera parameter and the second camera parameter;

[0022] generating a second plane optimization function using the plane data, the relative camera parameters, the first camera parameters, and the second camera parameters;

[0023] The plane data is optimized using the second plane optimization function.

[0024] Wherein, obtaining the three-dimensional coordinate points of the map based on the image data and the inertial data includes:

[0025] Using a front end to process the image data and the inertial data to obtain a real-time first three-dimensional coordinate point;

[0026] Transmitting the image data and inertial data of the front end to the back end;

[0027] The back end is used to process the image data and inertial data of the front end to obtain a non-real-time second three-dimensional coordinate point.

[0028] Wherein, generating plane data by using the three-dimensional coordinate points includes:

[0029] generating real-time first plane data based on the first three-dimensional coordinate points using the front end;

[0030] generating non-real-time second plane data based on the second three-dimensional coordinate points using the backend;

[0031] The map is constructed by combining the three-dimensional coordinate points and the plane data, including

[0032] The map is constructed by combining the first three-dimensional coordinate points, the first plane data, the second three-dimensional coordinate points, and the second plane data.

[0033] The step of constructing the map by combining the first three-dimensional coordinate point, the first plane data, the second three-dimensional coordinate point, and the second plane data includes:

[0034] Creating a three-dimensional grid using the first three-dimensional coordinate points and the second three-dimensional coordinate points;

[0035] generating a plurality of map planes using the first plane data, the second plane data, and the three-dimensional grid;

[0036] A final map is obtained using the plurality of map planes.

[0037] After generating a plurality of map planes, the map construction method further includes:

[0038] Based on the positions of the plurality of map planes in the three-dimensional grid, establishing associations between the first three-dimensional coordinate point, the second three-dimensional coordinate point, and the map planes, and establishing associations between adjacent map planes;

[0039] The obtaining of a final map by using the plurality of map planes includes:

[0040] A final map is constructed using the associations between the first three-dimensional coordinate point, the second three-dimensional coordinate point and the map plane, and the associations between adjacent map planes.

[0041] Another technical solution adopted by the present application is to provide a map construction system, the map construction system comprising:

[0042] A front end is configured to receive image data and inertial data, obtain real-time first three-dimensional coordinate points based on the image data and the inertial data, and generate first plane data based on the first three-dimensional coordinate points;

[0043] a backend, configured to receive the image data and the inertial data from the frontend, acquire non-real-time second three-dimensional coordinate points based on the image data and the inertial data, and generate second plane data based on the second three-dimensional coordinate points;

[0044] A plane module is used to construct the map by combining the first three-dimensional coordinate points, the first plane data, the second three-dimensional coordinate points and the second plane data.

[0045] The map construction system further includes:

[0046] A map module is used to store the map.

[0047] Another technical solution adopted by the present application is to provide a terminal device, the terminal device comprising a memory and a processor coupled to the memory;

[0048] The memory is used to store program data, and the processor is used to execute the program data to implement the map construction method as described above.

[0049] Another technical solution adopted by the present application is to provide a computer storage medium, wherein the computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the map construction method as described above.

[0050] The beneficial effects of this application are as follows: the terminal device provided by this application receives image data and inertial data; obtains three-dimensional coordinate points of the map based on the image data and inertial data; generates plane data using the three-dimensional coordinate points; and constructs the map by combining the three-dimensional coordinate points and the plane data. The map construction method of this application achieves tight coupling of image data, inertial data, and plane data, effectively improving environmental positioning through plane data, and providing accurate short-term tracking through inertial data, thereby achieving real-time and efficient pose and map estimation in weakly textured environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 This is a schematic diagram of the framework of an embodiment of a map construction system provided by this application;

[0053] Figure 2 This is a schematic diagram of the framework of another embodiment of the map construction system provided by the present application;

[0054] Figure 3 It is a schematic diagram of the structure of the homography factor provided in this application;

[0055] Figure 4 is a schematic diagram of the structure of the compressed homography factor provided by this application;

[0056] Figure 5 This is a flowchart of an embodiment of a map construction method provided by this application;

[0057] Figure 6 This is a schematic structural diagram of an embodiment of a terminal device provided by this application;

[0058] Figure 7 is a structural diagram of another embodiment of the terminal device provided by this application;

[0059] Figure 8 It is a structural diagram of an embodiment of a computer storage medium provided by this application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0061] For details, please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of the framework of an embodiment of the map construction system provided by this application. Figure 2 It is a schematic diagram of the framework of another embodiment of the map construction system provided by this application.

[0062] like Figure 1 As shown, the map construction system 100 provided by the present application includes a front end 11, a back end 12 and a plane module 13. Figure 2 In the system shown, the mapping system 100 may further include a mapping module 14 .

[0063] The front end 11 is configured to receive image data and inertial data, acquire real-time first three-dimensional coordinate points based on the image data and the inertial data, and generate first plane data based on the first three-dimensional coordinate points.

[0064] The back end 12 is configured to receive the image data and inertial data from the front end, obtain non-real-time second three-dimensional coordinate points based on the image data and the inertial data, and generate second plane data based on the second three-dimensional coordinate points.

[0065] The back end 12 is further configured to obtain a camera pose based on the image data and the inertial data, and optimize the second plane data using the camera pose.

[0066] The plane module 13 is used to construct the map based on the first three-dimensional coordinate point, the second three-dimensional coordinate point, the optimized first plane data, and the optimized second plane data.

[0067] The map module 14 is used to store the map, as well as key frames, maps and / or three-dimensional coordinate points in the image data.

[0068] The following is a detailed introduction to the above system components:

[0069] The front-end 11 primarily involves feature point extraction and tracking, IMU (Inertial Measurement Unit) pre-score, and real-time pose estimation based on a sliding window. The input data for the front-end 11 is RGB data (image color data), depth data (image depth data), and IMU data (inertial data). The final output is the pose (camera pose) and 3D point coordinates within the sliding window. These results serve as input to the back-end 12 and the planar module 13.

[0070] Due to the natural complementarity between IMUs and vision, VIO (Visual Inertial Odometry) has become a research hotspot in the field of SLAM in recent years. Even if the visual odometry loses tracking due to extreme issues such as drastic changes in illumination or texture loss, the IMU can provide good positioning results in a short period of time, effectively improving the robustness of the visual odometry. MSCKF and VINS-Mono are two typical representatives of VIO systems based on graph optimization and filtering, respectively. The graph optimization-based solution requires iterative steps, which will continuously integrate the IMU, resulting in a sharp increase in computational complexity. Therefore, the theory of IMU pre-integration was proposed. IMU pre-integration converts absolute measurements between frames into relative measurements, effectively solving this problem.

[0071] The VIO of this application tightly integrates RGB data, Depth data and IMU data, fuses all the above measurement values ​​using an inverse square filter, and adds the depth information of the Depth data to the visual measurement.

[0072] Specifically, the front end 11 needs to detect feature points in all captured images, such as ORB feature points, and calculate their descriptors. ORB feature points use the FAST (features from accelerated segment test) algorithm to detect feature points. The core idea of ​​FAST is to find points that stand out from the crowd. That is, a point is compared with its surrounding points. If it is different from most of the surrounding points, it is considered a feature point.

[0073] When the front-end 11 receives a new image, it uses KLT to track the feature points from the last image to the current image using the descriptors. It then projects the feature points containing 3D information onto the current image and uses the Hamming distance to select the closest projected feature point on the current image. The front-end 11 uses the closest projected feature point as the initial value to match the remaining feature points until the optimal observation point is found. Finally, the front-end 11 removes outliers using a RANdom Sampling Consensus (RANSAC) algorithm.

[0074] The plane module 13 primarily involves plane extraction, expansion and merging, and point-to-plane association. It uses Delaunay triangulation to achieve these functions. The input to the plane module 13 is the poses and 3D points from the front-end and back-end modules, as well as the depth data. The output is the plane data in the map.

[0075] Once a set of plane data is detected, the plane module 13 uses the data from the front-end 11 and the back-end 12 to expand the plane and associate the plane with a map landmark. Specifically, the plane module 13 uses Delaunay triangulation to create a 3D mesh and uses histograms to detect planes. In the embodiment of the present application, the plane module 13 detects vertical and horizontal planes. It can distinguish between vertical and horizontal planes by detecting whether the plane's grid normal within the 3D mesh is perpendicular or parallel.

[0076] Furthermore, the plane module 13 can also improve the accuracy of the plane by the following method: when detecting the plane from the histogram, the plane module 13 uses the image data and the three-dimensional plane points in the histogram to refine the parameters of the plane instead of directly using the scale value of the histogram.

[0077] For example, for a horizontal plane, suppose n = [0,0,1] T , the plane distance d is the average value of the Z axis value of the plane point; for the vertical plane, assuming n = [n x ,n y ,0] T, where the plane parameters of the vertical plane can be refined by the following formula:

[0078]

[0079] Where n is the number of plane points, is the position of the kth coordinate in the world coordinate system,

[0080] In this application, QR decomposition is used to solve the above equation (1). When plane parameters are detected, the plane module 13 can associate them with the three-dimensional grid through angles and distances.

[0081] Furthermore, the plane module 13 is further configured to expand a plurality of planes within the three-dimensional grid according to the optimized plane data, and to establish associations between planes and between planes and three-dimensional coordinate points.

[0082] Specifically, the plane module 13 checks whether a plane and another plane meet a certain angle and distance. When the angle between the two planes is less than a preset angle threshold and the distance is less than a first preset distance threshold, the two planes can be merged. Among them, the preset angle threshold in the embodiment of the present application can be set to 10°, and the first preset distance threshold can be set to 10cm. It should be noted that the above plane merging process can occur during the detection of a new plane or during the adjustment of an existing plane.

[0083] Specifically, the plane module 13 can also use a three-dimensional grid to associate more map points with a plane. When a three-dimensional grid is associated with a plane, the plane module 13 obtains the two-dimensional grid of the plane. If the two-dimensional coordinates of the map point are all within the two-dimensional grid and the distance from the map point to the plane is less than a second preset distance threshold, such as 10 cm, the plane module 13 adds the map point to the candidate set associated with the plane. If a map point in the candidate set is observed in more than three keyframes, the plane module 13 further checks the geometric consistency of the map point. The plane module 13 can force the map point to be associated with the plane by calculating the reprojection error from one keyframe to the plane; then, it calculates the reprojection error from another keyframe to the plane. If the two reprojection errors are close and the maximum reprojection error is less than a certain threshold, the map point is considered to be a planar point within the plane. If a map point fails the geometric consistency check multiple times, it is deleted from the candidate point set.

[0084] The backend 12 includes keyframe selection, local bundle adjustment optimization, loop closure detection and pose optimization, and global bundle adjustment optimization. The input to the backend 12 is the pose data, depth data, IMU data, and plane data from the frontend 11, and the output is a consistent map module 14 for the entire system.

[0085] Since the depth image is valid, the backend 12 integrates the depth information into the visual point feature measurement. The projection and depth residual at the i-th keyframe are defined as:

[0086]

[0087]

[0088] in, is the three-dimensional coordinate of the lth feature point of the i-th key frame, They are the rotation and translation from the inertial frame to the image frame respectively. Proj() is the function that projects the observation point from the camera coordinate system to the image coordinate system. is the lth feature observed in the i-th keyframe, z() is the third classification of this vector, and λ is the depth of the keyframe.

[0089] When a 3D map point is associated with a plane to be optimized, the terminal device can force the 3D map point to fall on the plane to be optimized, thereby converting the 3D map point into a normalized point. Therefore, instead of using the distance constraint between the common point and the plane, the terminal device can use a homography matrix to constrain the two keyframes and the plane to be optimized.

[0090] Specifically, assuming that the observation point on the plane to be optimized is π W The first camera coordinate system i of the first camera is th and the second camera coordinate system j of the second camera th Observed, from this, we can write the following point to plane reprojection equation:

[0091]

[0092]

[0093] Among them, the embodiment of this application defines p i =(x i ,y i ,1) T and p j =(x j ,y j ,1) T . Among them, (x i ,yi ,1) T =K -1 (u i ,v i ,1) T The coordinate point in the image coordinate system captured by the first camera to the camera coordinate system i th The mapping of the coordinate point, K is the eigenvalue matrix, (u i ,v i ) is the two-dimensional image feature in the key frame captured by the first camera, λ is the depth information of the image, and s is an unknown scale parameter.

[0094] Combined with the above formula, the projection equation of an observation point can be expressed as the following formula:

[0095]

[0096] Among them, W represents the world coordinate system, C represents the camera coordinate system, i and j are two different cameras, p i =(x i ,y i 1) is the normalized point in the camera coordinate system of the first camera, p j =(x j ,y j 1) is the coordinate of the normalized point in the camera coordinate system of the second camera, is the rotation corresponding to the first camera, is the rotation corresponding to the second camera, is the position corresponding to the first camera, is the position corresponding to the second camera; the plane to be optimized π=(n π ,d π ) are the direction and distance of the plane respectively. In addition, H is the homography matrix, I is the identity matrix, and s is a scale parameter.

[0097] Among them, the homography matrix H includes the camera parameters of the first key frame and The camera parameters of the second keyframe and And the plane parameters of the plane to be optimized and

[0098] Specifically, when the observation points are on the plane to be optimized, homography constraints and reprojection constraints are equivalent, where homography constraints do not require the 3D position of the point features. In the bundle adjustment (BA) problem, this application converts reprojection constraints into homography constraints, which is equivalent to removing many state variables of the observation points on the plane to be optimized. Finally, by utilizing a smaller and sparser Hessian matrix, the efficiency of bundle adjustment is greatly improved.

[0099] In the embodiment of the present application, the homography constraint equation of the observation point can be obtained from the projection equation in the above formula (6), as follows:

[0100]

[0101] Furthermore, the present application can eliminate the unknown scale parameter s from the above formula (7), thereby obtaining the homography cost function as follows:

[0102]

[0103]

[0104] Among them, C l is the coordinate matrix of the observation points.

[0105] In an embodiment of the present application, homography links two key frames and the state of a plane to be optimized. These three states may have many common observation results. Therefore, these observation results can be merged into one observation result to further improve the optimization speed.

[0106] Assuming that there are N point features on the plane to be optimized, the total homography cost function of the N point features on the plane to be optimized can be expressed as:

[0107]

[0108] in, To ensure the stability of the solution, this application uses eigenvalue decomposition, G h During the optimization process, it is a constant 9×9 matrix that depends only on constant values. Therefore, this matrix can be calculated in advance.

[0109] In the embodiment of the present application, the present application can combine the observations of N point features on the plane to be optimized into one observation matrix G h In, such as Figure 3 and Figure 4 , Figure 3 is the homography factor before merging, Figure 4The present application can effectively improve the efficiency of bundle adjustment by combining multiple cost functions into one cost function.

[0110] Furthermore, the Jacobian matrix of the compressed homography cost function, that is, the total homography cost function, can be defined as:

[0111]

[0112] Among them, L h is a lower triangular matrix, is an upper triangular matrix, L h and From the above constant 9×9 matrix G h Obtained through matrix decomposition.

[0113] Among them, the residual function, that is, the homography cost function is:

[0114]

[0115] By solving the Jacobian matrix of the total homography cost function, the optimization parameters of the plane to be optimized can be obtained.

[0116] In addition, for some plane points that are known to have point-surface constraints, such as internal observation points, this application can also use reprojection observations that delete these plane points, thereby reducing the dimension of the entire optimization state quantity and optimization matrix, which can further reduce the amount of calculation.

[0117] Furthermore, homography measurement relies on feature matching of map points and is easily affected by ambient lighting and texture. Therefore, this application adds geometric constraints to address this issue.

[0118] Specifically, the backend 12 uses a method of associating a single-frame point cloud with a plane to improve the accuracy of plane estimation and the stability of motion estimation in textureless scenes. The backend 12 uses a compressed point-to-plane cost function. is the lth plane observed in the i-th keyframe The set of N observation points above, yes The k-th three-dimensional coordinate point in . The residual from the k-th plane point to the plane can be defined as:

[0119]

[0120] in, is the camera pose of the i-th key frame, r ilk (X) is only one-dimensional, so the cost function of N observation points on the lth plane to the plane can be expressed as:

[0121]

[0122] Among them, G P is a 4×4 constant matrix that depends only on the observation value, not on the state. The Jacobian matrix of the new cost function can be expressed as:

[0123]

[0124] And the corresponding residual function is:

[0125]

[0126] Furthermore, because the map construction system 100 provided by this application incorporates IMU data, gravity is observable in the map construction system 100. Therefore, the backend 12 can determine whether a plane is horizontal or vertical based on the plane normal and the direction of gravity. If the i-th plane is horizontal and the j-th plane is vertical, the residual can be calculated as:

[0127]

[0128]

[0129] When a new keyframe is inserted into the mapping, the backend 12 performs LBA (local plane and point bundle adjustment) optimization. The camera poses, IMU data, and points and planes observed by the latest K keyframes are optimized through LBA. The other keyframes that observe these points and planes remain fixed in the LBA optimization. In an embodiment of the present application, K can be set to 20. The backend 12 uses the LM (Levenberg-Marquardt, least squares optimization) algorithm to solve the minimization problem, sets the maximum number of iterations to 10, and the maximum solution time to 0.2s.

[0130] Backend 12 integrates efficient plane constraints into local / global bundle adjustment optimization, while removing a large number of plane point reprojection constraints, thereby enabling fast and efficient optimization of map state quantities. The overall optimization formula, namely the first plane optimization function, is expressed as follows:

[0131]

[0132] in, Σ IMU is the constraint and covariance of the IMU; Σ C is the image reprojection error and covariance; and is the depth constraint and covariance; Σ CH It is based on point-surface constraints and covariance of homography; Σ CPP It is based on the point-surface constraints and covariance of 3D ToF points.

[0133] After the pose planar graph is optimized, all states need to be updated, including keyframe poses, IMU poses, points, and planes. For the GBA (global plane and point bundle adjustment) problem, the backend 12 integrates all measurements from the map construction system 100 in a tightly coupled manner, including IMU pre-integration, reprojection, compressed homography, compressed points to planes, and prior planes. Therefore, the backend 12 also introduces plane adjustment in loop detection and pose optimization. The optimization formula, namely the second plane optimization function, is expressed as follows:

[0134]

[0135] Among them, W is the world coordinate system, C is the camera coordinate system; N is the number of key frames, M is the number of planes, and m and n are loop frames. are the key frame poses at time i and i+1 respectively, is the relative position of the two. They are the lth plane in the world coordinate system and the camera coordinate system respectively.

[0136] For the minimization problem of the above optimization function, the backend 12 can also use the LM algorithm to solve it, and the maximum number of iterations can be set to 100 and the maximum solution time can be set to 2 seconds.

[0137] Please continue reading Figure 5 , Figure 5 This is a flow chart of an embodiment of the map construction method provided by this application. Figure 1 and / Figure 2 The map construction system shown is not described in detail herein.

[0138] like Figure 5 As shown, the map construction method of the embodiment of the present application specifically includes the following steps:

[0139] Step S51: receiving image data and inertial data.

[0140] In an embodiment of the present application, the front end receives image data and inertial data and transmits the image data and inertial data to the back end. The image data can be collected by a camera device, and the inertial data can be collected by an inertial sensor.

[0141] Step S52: Based on the image data and inertial data, obtain the three-dimensional coordinate points of the map.

[0142] In an embodiment of the present application, the terminal device can fit the three-dimensional coordinate points in the map based on the image data and inertial data.

[0143] For example, in a map-building system, the front-end directly acquires image data from a camera device and inertial data from an inertial sensor, while the back-end acquires image data and inertial data from the front-end. Both the front-end and the back-end need to generate three-dimensional coordinate points based on the image and inertial data, and fit several planes using the three-dimensional coordinate points. The difference between the front-end and the back-end is that the front-end processes image and inertial data in real time, while the back-end processes them non-real time. Specifically, the front-end can promptly fit real-time three-dimensional coordinate points and planes based on the input image and inertial data, with a fast fitting speed that can meet the real-time map-building requirements of the map-building system. The back-end needs to receive the image data and inertial data transmitted by the front-end, and then fit them into non-real-time three-dimensional coordinate points and planes. Because the back-end is responsible for non-real-time tasks, the fitting results for the three-dimensional coordinate points and planes are more accurate than those of the front-end.

[0144] In this embodiment of the present application, the front end obtains real-time first three-dimensional coordinate points based on image data and inertial data; the back end obtains non-real-time second three-dimensional coordinate points based on the front end image data, inertial data, and the first three-dimensional coordinate points. The front end generates first plane data based on the first three-dimensional coordinate points; and the back end generates second plane data based on the second three-dimensional coordinate points.

[0145] Step S53: Generate plane data using three-dimensional coordinate points.

[0146] In the embodiment of the present application, the terminal device uses three-dimensional coordinate points to fit a plurality of planes, i.e., to generate plane data. After generating the plane data, the terminal device can also optimize the plane data using the constraints determined by the image data and the relationship data, so that the optimized plane data is more accurate.

[0147] Specifically, in the map construction system, the backend uses the inertial data to obtain inertial constraints; uses the plane data and the three-dimensional coordinate points to obtain reprojection errors and depth constraints; generates a first plane optimization function according to the inertial constraints, the reprojection errors and the depth constraints; and uses the first plane optimization function to optimize the plane data.

[0148] Among them, the above-mentioned constraints such as inertia constraint, reprojection error, depth constraint, etc. can all be used as constraints in plane optimization, so that during the plane optimization process, the plane data can be continuously iteratively optimized within a limited range, and the most accurate plane data can be optimized at the lowest iteration cost. It should be noted that the first plane optimization function of the embodiment of the present application may include constraints formed by a combination of one or more of the above, that is, the terminal device can select one or more, or even all of the constraints from the above to form a plane optimization function to optimize the plane data.

[0149] Specifically, in one embodiment, the expression of the first plane optimization function can be expressed as:

[0150]

[0151] in, Σ IMU is the constraint and covariance of the IMU; Σ C is the image reprojection error and covariance; and is the depth constraint and covariance.

[0152] Furthermore, the terminal device may also add other constraints, such as point-plane constraints, to the first plane optimization function.

[0153] For example, in another embodiment, the back end can also obtain the first coordinate of the observation point associated with the plane in the first key frame and the second coordinate of the observation point in the second key frame based on the plane data and the three-dimensional coordinate point; use the first coordinate, the second coordinate and the preset projection equation to obtain the point-surface constraint between the observation point and the plane; generate a first plane optimization function according to the point-surface constraint, the inertia constraint, the reprojection error and the depth constraint.

[0154] Specifically, the terminal device can determine the point-surface constraints of the plane by calculating the point-surface relationship between the observation point and the plane to be optimized, and use the point-surface constraints as constraints of the first plane optimization function. This can increase the constraint types of the first plane optimization function and further improve the plane optimization effect.

[0155] Among them, after the first plane optimization function introduces point-surface constraints, the expression can be expressed as:

[0156]

[0157] in, Σ IMU is the constraint and covariance of the IMU; Σ C is the image reprojection error and covariance; and is the depth constraint and covariance; Σ CH It is based on point-surface constraints and covariance of homography; Σ CPP It is based on the point-surface constraints and covariance of 3D ToF points.

[0158] In addition to optimizing the plane through constraints, the terminal device of the embodiment of the present application can also optimize the plane through information such as camera posture.

[0159] Specifically, in the map construction system, the backend can also obtain the first camera parameters for capturing the first key frame and the second camera parameters for capturing the second key frame based on the image data; obtain the relative camera parameters of the first camera parameters and the second camera parameters; generate a second plane optimization function using the plane data, the relative camera parameters, the first camera parameters and the second camera parameters; and optimize the plane data using the second plane optimization function.

[0160] In one embodiment, the expression of the second plane optimization function can be expressed as:

[0161]

[0162] Among them, W is the world coordinate system, C is the camera coordinate system; N is the number of key frames, M is the number of planes, and m and n are loop frames. are the key frame poses at time i and i+1, namely the first camera parameters and the second camera parameters, is the relative pose of the two, that is, the relative camera parameters, They are the lth plane in the world coordinate system and the camera coordinate system respectively.

[0163] It should be noted that the optimization process of the back-end for plane data is specifically introduced in the description of the above embodiment and formula (18) and formula (19), and will not be repeated here.

[0164] Step S54: combining the three-dimensional coordinate points and the plane data to construct a map.

[0165] Finally, the terminal device integrates the first three-dimensional coordinate points and the first plane data of the front end, the second three-dimensional coordinate points of the back end, and the optimized second plane data to construct the final map.

[0166] Specifically, the plane module in the map construction system uses the first three-dimensional coordinate points and the second three-dimensional coordinate points to create a three-dimensional grid; and uses the first plane data, the second plane data and the three-dimensional grid to generate a number of map planes. The plane module forms a three-dimensional grid by connecting and associating the first three-dimensional coordinate points of the front end and the second three-dimensional coordinate points of the back end, and then fits a number of map planes in the three-dimensional grid according to the first plane data of the front end and the second plane data of the back end. It should be noted that in addition to the coordinate points of the above-mentioned three-dimensional coordinate points that are themselves within the map plane, other coordinate points can also be forcibly associated with the nearest map plane, so that in the final map construction process, the map can be constructed only by the map plane, which can effectively reduce the computing resources consumed by the three-dimensional coordinate points in the map construction process and improve the efficiency of map construction.

[0167] Specifically, the plane module establishes an association between the first three-dimensional coordinate point, the second three-dimensional coordinate point and the map plane, and establishes an association between adjacent map planes based on the positions of the several map planes in the three-dimensional grid; and uses the association between the first three-dimensional coordinate point, the second three-dimensional coordinate point and the map plane, and the association between adjacent map planes to construct the final map.

[0168] For example, the plane module checks whether a plane and another plane meet a certain angle and distance. When the angle between the two planes is less than a preset angle threshold and the distance is less than a first preset distance threshold, the two planes can be merged. Among them, the preset angle threshold in the embodiment of the present application can be set to 10°, and the first preset distance threshold can be set to 10cm. It should be noted that the above plane merging process can occur during the detection of a new plane or during the adjustment of an existing plane.

[0169] For another example, the plane module can also use a three-dimensional grid to associate more map points with the plane. When a three-dimensional grid is associated with a plane, the plane module obtains the two-dimensional grid of the plane. If the two-dimensional coordinates of the map point are all within the two-dimensional grid and the distance from the map point to the plane is less than a second preset distance threshold, such as 10 cm, the plane module adds the map point to the candidate set associated with the plane. If a map point in the candidate set is observed in more than three keyframes, the plane module further checks the geometric consistency of the map point. The plane module can force the map point to be associated with the plane by calculating the reprojection error of the map point from one keyframe to the plane; then, it calculates the reprojection error from another keyframe to the plane. If the two reprojection errors are close and the maximum reprojection error is less than a certain threshold, the map point is considered to be a plane point within the plane. If a map point fails the geometric consistency check multiple times, it is deleted from the candidate point set.

[0170] The above embodiment is only one common case of the present application and does not limit the technical scope of the present application. Therefore, any minor modifications, equivalent changes or modifications made to the above content based on the essence of the present application solution are still within the scope of the technical solution of the present application.

[0171] Please continue to see Figure 6 , Figure 6 FIG. 3 is a schematic diagram of the structure of an embodiment of a terminal device provided by the present application, wherein the terminal device 30 includes a receiving module 31 , a planar module 32 and a construction module 33 .

[0172] The receiving module 31 is configured to receive image data and inertial data.

[0173] The plane module 32 is configured to obtain three-dimensional coordinate points of the map based on the image data and inertial data, and generate plane data using the three-dimensional coordinate points.

[0174] The construction module 33 is used to construct the map by combining the three-dimensional coordinate points and the plane data.

[0175] The plane module 32 is further used to obtain inertial constraints using the inertial data; obtain reprojection errors and depth constraints using the plane data and the three-dimensional coordinate points; generate a first plane optimization function according to the inertial constraints, the reprojection errors and the depth constraints; and optimize the plane data using the first plane optimization function.

[0176] The plane module 32 is also used to obtain the first coordinate of the observation point associated with the plane in the first key frame and the second coordinate of the observation point in the second key frame based on the plane data and the three-dimensional coordinate point; use the first coordinate, the second coordinate and the preset projection equation to obtain the point-surface constraint between the observation point and the plane; and generate a first plane optimization function according to the point-surface constraint, the inertia constraint, the reprojection error and the depth constraint.

[0177] The plane module 32 is further used to obtain, based on the image data, first camera parameters for capturing a first key frame and second camera parameters for capturing a second key frame; obtain relative camera parameters of the first camera parameters and the second camera parameters; generate a second plane optimization function using the plane data, the relative camera parameters, the first camera parameters, and the second camera parameters; and optimize the plane data using the second plane optimization function.

[0178] The plane module 32 is further configured to process the image data and the inertial data using the front end to obtain a first three-dimensional coordinate point in real time; transmit the image data and the inertial data of the front end to the back end; and process the image data and the inertial data of the front end using the back end to obtain a second three-dimensional coordinate point in non-real time.

[0179] The plane module 32 is also used to generate real-time first plane data based on the first three-dimensional coordinate points using the front end; generate non-real-time second plane data based on the second three-dimensional coordinate points using the back end; and construct the map by combining the three-dimensional coordinate points and the plane data, including constructing the map by combining the first three-dimensional coordinate points, the first plane data, the second three-dimensional coordinate points and the second plane data.

[0180] The construction module 33 is further used to create a three-dimensional grid using the first three-dimensional coordinate points and the second three-dimensional coordinate points; generate several map planes using the first plane data, the second plane data and the three-dimensional grid; and obtain a final map using the several map planes.

[0181] The construction module 33 is further used to establish an association between the first three-dimensional coordinate point, the second three-dimensional coordinate point and the map plane based on the positions of the multiple map planes in the three-dimensional grid, and to establish an association between adjacent map planes; and to construct a final map using the association between the first three-dimensional coordinate point, the second three-dimensional coordinate point and the map plane, and the association between adjacent map planes.

[0182] Please continue to see Figure 7 , Figure 7 1 is a schematic diagram of the structure of another embodiment of the terminal device provided by the present application. The terminal device 500 of the embodiment of the present application includes a processor 51, a memory 52, an input / output device 53 and a bus 54.

[0183] The processor 51 , the memory 52 , and the input / output device 53 are respectively connected to a bus 54 . The memory 52 stores program data, and the processor 51 is used to execute the program data to implement the map construction method described in any of the above embodiments.

[0184] In the embodiment of the present application, the processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or the processor 51 may be any conventional processor.

[0185] This application also provides a computer storage medium, please continue to refer to Figure 8 , Figure 8 6 is a schematic structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores program data 61. When the program data 61 is executed by a processor, it is used to implement the map construction method of any of the above embodiments.

[0186] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0187] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Equivalent structures or equivalent process changes made by utilizing the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A map construction method, characterized in that: include: Receive image data and inertial data; Acquiring three-dimensional coordinate points of the map based on the image data and the inertial data; generating plane data using the three-dimensional coordinate points; Combining the three-dimensional coordinate points and the plane data to construct the map; The obtaining of three-dimensional coordinate points of the map based on the image data and the inertial data includes: Using a front end to process the image data and the inertial data to obtain a real-time first three-dimensional coordinate point; Transmitting the image data and inertial data of the front end to the back end; Using the back end to process the image data and inertial data of the front end to obtain a second three-dimensional coordinate point in non-real time; The step of combining the three-dimensional coordinate points and the plane data to construct the map includes: Connecting and associating the first three-dimensional coordinate points with the second three-dimensional coordinate points to form a three-dimensional grid; A plurality of map planes are fitted in the three-dimensional grid according to the plane data.

2. The map construction method according to claim 1, characterized in that: After generating plane data using the three-dimensional coordinate points, the map construction method further includes: obtaining an inertial constraint using the inertial data; Obtaining a reprojection error and a depth constraint using the plane data and the three-dimensional coordinate points; generating a first plane optimization function according to the inertia constraint, the reprojection error, and the depth constraint; The plane data is optimized using the first plane optimization function.

3. The map construction method according to claim 2, characterized in that: Generating a first plane optimization function according to the inertia constraint, the reprojection error, and the depth constraint includes: Based on the plane data and the three-dimensional coordinate point, obtaining a first coordinate of an observation point associated with the plane in a first key frame, and a second coordinate of the observation point in a second key frame; Obtaining a point-plane constraint between the observation point and the plane using the first coordinate, the second coordinate, and a preset projection equation; A first plane optimization function is generated according to the point-plane constraint, the inertia constraint, the reprojection error, and the depth constraint.

4. The map construction method according to any one of claims 1 to 3, characterized in that: After generating plane data using the three-dimensional coordinate points, the map construction method further includes: Based on the image data, obtaining first camera parameters for capturing a first key frame and second camera parameters for capturing a second key frame; Obtaining relative camera parameters of the first camera parameter and the second camera parameter; generating a second plane optimization function using the plane data, the relative camera parameters, the first camera parameters, and the second camera parameters; The plane data is optimized using the second plane optimization function.

5. The map construction method according to claim 1, characterized in that: The generating of plane data by using the three-dimensional coordinate points includes: generating real-time first plane data based on the first three-dimensional coordinate points using the front end; generating non-real-time second plane data based on the second three-dimensional coordinate points using the backend; The step of combining the three-dimensional coordinate points and the plane data to construct the map includes: The map is constructed by combining the first three-dimensional coordinate points, the first plane data, the second three-dimensional coordinate points, and the second plane data.

6. The map construction method according to claim 5, characterized in that: The step of constructing the map by combining the first three-dimensional coordinate point, the first plane data, the second three-dimensional coordinate point, and the second plane data includes: Creating a three-dimensional grid using the first three-dimensional coordinate points and the second three-dimensional coordinate points; generating a plurality of map planes using the first plane data, the second plane data, and the three-dimensional grid; A final map is obtained using the plurality of map planes.

7. The map construction method according to claim 6, characterized in that: After generating a plurality of map planes, the map construction method further includes: Based on the positions of the plurality of map planes in the three-dimensional grid, establishing associations between the first three-dimensional coordinate point, the second three-dimensional coordinate point, and the map planes, and establishing associations between adjacent map planes; The obtaining of a final map by using the plurality of map planes includes: A final map is constructed using the associations between the first three-dimensional coordinate point, the second three-dimensional coordinate point and the map plane, and the associations between adjacent map planes.

8. A map construction system, characterized in that: The map construction system includes: A front end is configured to receive image data and inertial data, obtain real-time first three-dimensional coordinate points based on the image data and the inertial data, and generate first plane data based on the first three-dimensional coordinate points; a backend, configured to receive the image data and the inertial data from the frontend, acquire non-real-time second three-dimensional coordinate points based on the image data and the inertial data, and generate second plane data based on the second three-dimensional coordinate points; a plane module, configured to construct the map by combining the first three-dimensional coordinate points, the first plane data, the second three-dimensional coordinate points, and the second plane data; The plane module is further configured to connect and associate the first three-dimensional coordinate points with the second three-dimensional coordinate points to form a three-dimensional grid; and to fit a plurality of map planes in the three-dimensional grid according to the plane data.

9. The map construction system according to claim 8, wherein: The map construction system further includes: A map module is used to store the map.

10. A terminal device, characterized in that: The terminal device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the map construction method according to any one of claims 1 to 7.

11. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the map construction method according to any one of claims 1 to 7.

Images