3D map reconstruction methods and equipment

Abstract

This application relates to the field of 3D map construction technology, specifically to a 3D map reconstruction method and device. The method generates initial sub-maps by acquiring multiple image data and performing feature extraction. Multiple initial sub-maps with overlapping perspectives are then locally sparsely reconstructed to generate multiple sub-maps. Global feature extraction is performed on these sub-maps to generate a global feature database, and similar image pairs are determined based on sub-map similarity. Corresponding feature points are determined by performing local feature matching on similar image pairs. For 3D-3D correspondences, i.e., when corresponding 3D feature points exist in two sub-maps, the system employs a rigid body transformation algorithm to accurately calculate the relative pose between the two sub-maps by minimizing the distance between 3D feature point pairs.

Description

Technical Field

[0001] This application relates to the field of 3D map construction technology, specifically to a 3D map reconstruction method and device. Background Technology

[0002] Mobile 3D scanning equipment is characterized by its portability, speed, and efficiency, and has been widely used in fields such as architectural surveying, cultural heritage protection, film and game production, and virtual reality. Its positioning and mapping capabilities directly affect the coordinate accuracy and structural reproduction of 3D data.

[0003] Among various sensors, camera-based visual sensors provide high-resolution texture information, which is key to achieving highly realistic modeling and visualization. However, in our research, we found that in actual 3D scanning, due to the complexity and unpredictability of scene structures, a single scan often fails to completely cover all detailed areas, especially in the presence of occlusion, reflections, or small structures. Initial scans may result in texture loss, geometric deformation, or insufficient detail. Summary of the Invention

[0004] In view of the above problems, this application provides a three-dimensional map reconstruction method and device to solve the above-mentioned technical problems existing in the prior art.

[0005] One aspect of this application provides a three-dimensional map reconstruction method, including:

[0006] Multiple image data are acquired, and features are extracted from the multiple image data. Multiple sub-maps are generated by local sparse reconstruction of multiple image data with overlapping viewpoints.

[0007] Global feature extraction is performed on multiple sub-maps to generate a global feature database;

[0008] Similar image pairs are determined from the global feature database based on the similarity of each of the sub-maps;

[0009] Local feature matching is performed on the similar image pairs to determine the corresponding feature points of the image data contained in the similar image pairs;

[0010] If the corresponding feature point has a corresponding 3D feature point in the respective sub-map coordinate system of the image data, then a rigid body transformation is performed on the 3D feature point to make the distance between the 3D feature points less than a preset first threshold, and the pose transformation matrix corresponding to the corresponding feature point is determined.

[0011] Based on the pose transformation matrix corresponding to the corresponding feature points, the image data contained in the similar image pairs are initially aligned in the global coordinate system to generate a fused global model.

[0012] The fused global model is then globally optimized to complete the reconstruction of the 3D map.

[0013] Preferably, in some embodiments, the step of acquiring multiple image data, extracting features from the multiple image data, and performing local sparse reconstruction of the multiple image data with overlapping viewpoints to generate multiple sub-maps includes:

[0014] The image data is preprocessed;

[0015] Feature points are extracted from the preprocessed image data, and a local sparse point cloud is constructed based on the feature points;

[0016] The pose data corresponding to the image data is obtained by local beam adjustment based on the local sparse point cloud.

[0017] Based on the pose data, multiple sub-maps are generated by locally sparse reconstruction of multiple image data with overlapping viewpoints.

[0018] Preferably, in some embodiments, the global feature database includes global feature descriptors of the image data contained in each sub-map;

[0019] The step of determining similar image pairs from the global feature database based on the similarity of each of the sub-maps includes:

[0020] Determine the global feature descriptor of the image data to be queried;

[0021] Determine the cosine similarity between the global feature descriptor of the image data to be queried and the global feature descriptor in the global feature database;

[0022] The similar image pairs contained in the sub-map are determined based on the cosine similarity.

[0023] Preferably, in some embodiments, the step of performing local feature matching on the similar image pairs to determine the corresponding feature points of the image data contained in the similar image pairs includes:

[0024] Extract local feature points and global feature descriptors from the image data contained in the similar image pairs;

[0025] Based on the local feature points and global feature descriptors, a nearest neighbor search algorithm is used to perform local feature matching on the image data contained in the similar image pairs;

[0026] Determine the corresponding feature points in the image data contained in the similar image pair.

[0027] Preferably, in some embodiments, the method further includes:

[0028] If the corresponding feature point has a 3D feature point and a corresponding 2D feature point in the respective sub-map coordinate system of the image data, then the reprojection error of the 3D feature point on the 2D feature point is determined, such that the reprojection error is less than a preset second threshold, and the pose transformation matrix corresponding to the corresponding feature point is determined.

[0029] Preferably, in some embodiments, the method further includes:

[0030] If the corresponding feature point has a corresponding 2D feature point in the respective sub-map coordinate system of the image data, then the fundamental matrix of the image data contained in the similar image pair is determined, or the essential matrix of the image data contained in the similar image pair is determined, so that the number of feature points satisfying the fundamental matrix equation is greater than a preset third threshold, or the number of feature points satisfying the essential matrix equation is greater than a preset fourth threshold, and the pose transformation matrix corresponding to the corresponding feature point is determined.

[0031] Preferably, in some embodiments, the step of performing global optimization on the fused global model to complete the reconstruction of the 3D map includes:

[0032] The fused global model is globally optimized using the global bundle adjustment method.

[0033] Preferably, in some embodiments, the global optimization of the fused global model using the global bundle adjustment method includes:

[0034] The sparse point cloud set corresponding to the image data contained in the similar image pair is decomposed, and its covariance matrix is ​​calculated.

[0035] The flatness of the sparse point cloud is determined based on the covariance matrix.

[0036] If the flatness is less than a preset flatness threshold, then a point-to-surface error term is determined, and the fused global model is globally optimized based on the point-to-surface error term.

[0037] Preferably, in some embodiments, the global optimization of the fused global model using the global bundle adjustment method includes:

[0038] Determine the common visual feature points among the image data contained in the similar image pairs to form a set of common visual feature points;

[0039] Randomly select a preset proportion of common-view feature points from the set of common-view feature points as fixed control points;

[0040] The fusion global model is globally optimized, and the preset ratio is reduced until all the common-view feature points are used as fixed control points.

[0041] According to another aspect of the embodiments of this application, a three-dimensional map reconstruction device is also proposed, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0042] The memory is used to store at least one program that causes the processor to perform the operation of the three-dimensional map reconstruction method described in the above embodiments.

[0043] The 3D map reconstruction method and device proposed in this application determine corresponding feature points by performing local feature matching on similar image pairs. For 3D-3D correspondence, that is, when there are corresponding 3D feature points in two sub-maps, the system adopts a rigid body transformation solution algorithm to accurately calculate the relative pose between the two sub-maps by minimizing the distance between the 3D feature point pairs. This method does not rely on external positioning devices such as GNSS / RTK, which reduces costs and deployment difficulty. It has a high degree of automation, and the entire process automatically completes sub-map alignment and fusion based on image features, without the need for manual point selection.

[0044] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0045] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0046] Figure 1 This is a schematic diagram of the structure of a three-dimensional map reconstruction device provided in an embodiment of this application;

[0047] Figure 2 A flowchart illustrating a three-dimensional map reconstruction method provided in an embodiment of this application;

[0048] Figure 3 This is another schematic diagram of a three-dimensional map reconstruction method provided in an embodiment of this application. Detailed Implementation

[0049] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0050] Among existing 3D map reconstruction technologies, one is the manual alignment method using image control points. This method involves deploying and acquiring image control points in the scene, manually selecting and aligning the coordinate system using a point-clicking method, and then performing global bundle adjustment. This method is highly dependent on the operator's experience, has a large workload, and is time-consuming. Another method is the automatic alignment method based on absolute coordinate information. This method uses external positioning equipment such as GNSS / RTK and total stations to acquire high-precision pose information, achieving automatic alignment of multiple data acquisitions. This method is more efficient, but requires expensive hardware support, has poor compatibility between different devices, and is difficult to integrate with multiple image data sources.

[0051] The inventors of this application discovered in their research that a reconstruction method based on image feature matching is also possible. This method extracts image feature points and reconstructs the 3D structure based on the matching relationships between feature points. However, when dealing with multi-source image data, the lack of an effective sub-map fusion mechanism often makes it difficult to achieve high-quality, globally consistent reconstruction. Therefore, existing technologies face problems such as low efficiency, reliance on external devices, and difficulty in achieving automatic alignment when processing multi-source image data. There is an urgent need for a 3D map reconstruction method that can efficiently process multi-source image data.

[0052] In view of this, the inventors of this application propose a 3D map reconstruction method and device to address the aforementioned problems. The method involves acquiring multiple image data and extracting features to generate initial sub-maps. Multiple initial sub-maps with overlapping perspectives are then locally sparsely reconstructed to generate multiple sub-maps. Global feature extraction is performed on these sub-maps to generate a global feature database, and similar image pairs are determined based on sub-map similarity. Local feature matching is performed on similar image pairs to determine corresponding feature points. For 3D-3D correspondences (i.e., when corresponding 3D feature points exist in two sub-maps), the system employs a rigid body transformation algorithm to accurately calculate the relative pose between the two sub-maps by minimizing the distance between the 3D feature point pairs. This method utilizes direct geometric constraints in 3D space, offering higher accuracy and robustness compared to 2D-3D or 2D-2D matching. It effectively handles situations with inconsistent scales and large perspective changes, significantly improving the accuracy of multi-source data fusion and achieving automatic alignment and fusion of multi-source image data without requiring additional hardware. In addition, the embodiments of this application also adopt a combination of global features and local features to improve the accuracy and robustness of feature matching, and introduce point-surface geometric constraints and a common-view feature point alternation optimization strategy to ensure the global consistency of the model after multi-source data fusion.

[0053] The 3D map reconstruction device proposed in this application embodiment can be a mobile 3D scanning device, such as a handheld scanning device, a vehicle-mounted scanning device, a mobile robot scanning device, etc., or a mobile phone, smart glasses or other smart device with scanning function. In this application embodiment, no limitation is made. Figure 1A schematic diagram of the structure of the three-dimensional map reconstruction device proposed in the embodiments of this application is shown, which is used to execute the three-dimensional map reconstruction method proposed in the embodiments of this application.

[0054] like Figure 1 As shown, the 3D map reconstruction device may include: a processor 102, a memory 106, a communication interface 104, and a communication bus 108.

[0055] The processor 102, memory 106, and communication interface 104 communicate with each other via communication bus 108. The memory 106 stores at least one program 110, which causes the processor 102 to execute steps related to the three-dimensional map reconstruction method proposed in this application embodiment.

[0056] Specifically, program 110 may include program code, which includes computer-executable instructions.

[0057] Processor 102 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the UAV direction determination device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0058] Memory 106 is used to store program 110. Memory 106 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0059] Specifically, program 110 can be called by processor 102 to cause the 3D map reconstruction device to execute the 3D map reconstruction method proposed in the embodiments of this application. Specifically, the 3D map reconstruction method is as follows: Figure 2 As shown, it includes:

[0060] Step S100: Acquire multiple image data and extract features from the multiple image data to generate an initial sub-map.

[0061] In the 3D map reconstruction method proposed in this invention, image data is the fundamental input of the entire reconstruction process. Its quality, diversity, and organization directly affect the accuracy and completeness of the reconstruction results. The image data processed in this embodiment typically comes from multiple different sources, including but not limited to drone aerial images, images acquired by ground mobile devices, images captured by fixed monitoring cameras, and historical image archives. These images may be acquired using different camera devices, with different resolutions, focal lengths, exposure parameters, and shooting angles. In practical applications, image data is usually divided into two main categories: panoramic overview images and local detail images. Panoramic overview images are usually acquired by high-altitude drones, covering a wide area and suitable for constructing the overall structure of a scene; local detail images are taken at close range by ground devices, containing rich texture and detail information, suitable for enhancing the local accuracy of the model.

[0062] In this step, when extracting features from multiple image data, the image data needs to be preprocessed first. The preprocessed image includes quality assessment, distortion correction, image resolution standardization, color normalization, and metadata extraction. The image quality assessment includes removing low-quality images that are blurry, underexposed or overexposed, or have excessive noise. Distortion correction includes correcting lens distortion based on camera calibration parameters, especially barrel distortion caused by wide-angle and fisheye lenses. Image metadata extraction includes parsing EXIF ​​information to obtain shooting time, GPS location, camera parameters, etc. Image resolution standardization includes adjusting images of different sizes to a uniform resolution range suitable for processing, in order to balance computational efficiency and detail preservation.

[0063] After preprocessing the image data, feature points are extracted from the preprocessed image data. Feature extraction and matching algorithms, such as SIFT or ORB, can be used to extract locally invariant feature points from the image. These locally invariant feature points are accompanied by feature descriptors. The more similar the feature descriptors are, the more invariant they are considered to be in the scene.

[0064] After extracting feature points, a local sparse point cloud is constructed based on the feature points, and the pose data corresponding to the image data is obtained through local bundle adjustment. The pose data includes the camera pose and three-dimensional point coordinates corresponding to the image data.

[0065] Based on the pose data, multiple image data with overlapping viewpoints are locally sparsely reconstructed to generate multiple sub-maps. In this step, each batch of image data with sufficiently overlapping viewpoints is treated as a sub-map, and local sparse reconstruction is performed independently. Sufficiently overlapping viewpoints refer to setting a threshold for the number of matching features between pairs of images; if the number is below this threshold, the images are considered disconnected and will be divided into new image subsets. Each sub-map stores one connected image subset.

[0066] Step S200: Perform global feature extraction on the multiple sub-maps to generate a global feature database;

[0067] For each image in all sub-maps, its global feature descriptor is extracted, and a global feature database is constructed. This process aims to compress high-dimensional image information into a fixed-length vector for rapid similarity calculation.

[0068] Let the first The first in the sub-map The image is Through a global feature extraction function (Such as NetVLAD or any advanced image retrieval model) to obtain its descriptor:

[0069] All descriptors are stored in an index structure (such as KD-Tree, HNSW, or Faiss) to form a global feature database.

[0070] Step S300: Determine similar image pairs from the global feature database based on the similarity of each of the sub-maps;

[0071] This step automatically retrieves cross-map matching candidate pairs, i.e., similar image pairs, with overlapping areas across different sub-maps based on the similarity of each sub-map. Global feature indexing is performed on the features of all sub-maps, using global feature description methods such as the bag-of-words model or NetVLAD. Identifying similar image pairs can also be understood as image recall. Image recall involves finding one or more similar images (query image) on an image-by-image basis. The similarity is determined by the feature descriptors, and the correct recalled image should capture the same scene (or a portion thereof) as the query image.

[0072] Assuming the image data to be queried is By calculating its global descriptor Similarity with all other descriptors in the global feature database The method retrieves the K most similar images. In this embodiment, cosine similarity is preferably used to determine the similarity between two images; alternatively, the reciprocal of the L2 distance can also be used to determine their similarity.

[0073] Step S400: Perform local feature matching on the similar image pairs to determine the corresponding feature points of the image data contained in the similar image pairs;

[0074] For each recalled image pair ,in, It is the result set of the recall, which contains image pairs across subgraphs. This requires fine-grained local feature matching and geometric verification to filter out false recalls and estimate relative pose.

[0075] When performing local feature matching on the similar image pairs, the local feature points and global feature descriptors of the image data contained in the similar image pairs are first extracted. The local feature points (such as SIFT, ORB, SuperPoint, etc.) and their global feature descriptors of the two images are extracted.

[0076] Based on the local feature points and global feature descriptors, a nearest neighbor search algorithm (such as FLANN) is used to perform local feature matching on the image data contained in the similar image pairs to obtain a preliminary set of 2D-2D corresponding point pairs. ,in and These are the corresponding feature points in the image data contained in the similar image pair.

[0077] After determining the corresponding feature points of the image data contained in the similar image pair, in this embodiment of the application, in order to improve the success rate and robustness of the expanded map matching, different geometric models are used for robustness estimation according to the information richness of the sub-image in which the similar image pair is located.

[0078] Step S500: Analyze the feature points and use different geometric models for robust estimation.

[0079] Specifically, there are three cases: if the corresponding feature point has a corresponding 3D feature point in the respective sub-map coordinate system of the image data, i.e. 3D-3D matching mode, then proceed to step S510.

[0080] If the corresponding feature point has a 3D feature point and a corresponding 2D feature point in the respective sub-map coordinate system of the image data, i.e., a 2D-3D matching mode, then proceed to step S520;

[0081] If the corresponding feature point has a corresponding 2D feature point in the respective sub-map coordinate system of the image data, i.e., a 2D-2D matching mode, then proceed to step S530;

[0082] Step S510: If the corresponding feature point has a corresponding 3D feature point in the respective sub-map coordinate system of the image data, then perform a rigid body transformation on the 3D feature point so that the distance between the 3D feature points is less than a preset first threshold.

[0083] If the feature points in two images have corresponding 3D points in their respective subgraph coordinate systems... and Then solve for a rigid body transformation This minimizes the distance between point pairs.

[0084] That is, to solve This problem can be solved using closed-form SVD. If the distance between two 3D points is less than a preset first threshold, the matching relationship between the corresponding feature points is considered verified. When two 3D feature points exist simultaneously, the problem is solved using rigid body transformation, which provides the strongest and most reliable constraint.

[0085] Step S520: If the corresponding feature point has a 3D feature point and a corresponding 2D feature point in the respective sub-map coordinate system of the image data, then determine the reprojection error of the 3D feature point on the 2D feature point, such that the reprojection error is less than a preset second threshold.

[0086] Known image data 3D points corresponding to feature points (in its local coordinate system), and they in 2D projection in At this point, we need to solve for a pose transformation. This minimizes the reprojection error.

[0087] That is, to solve This minimizes the reprojection error of 3D feature points onto 2D feature points, meaning the reprojection error is less than a preset second threshold. It is the camera projection function.

[0088] Step S530: If the corresponding feature points have corresponding 2D feature points in their respective sub-map coordinate systems of the image data, i.e. 2D-2D matching, then the number of interior points satisfying the fundamental matrix F or the essential matrix E can be maximized by calculating the fundamental matrix equation or the essential matrix equation.

[0089] The fundamental matrix (F) or essential matrix (E) is used to describe the geometric relationship between corresponding points in two images.

[0090] The fundamental matrix F is a 3x3 matrix used to describe the epipolar geometry between two unlabeled images. For corresponding points in the two images... and They satisfy the following equation:

[0091]

[0092] in, and These are normalized planar coordinates. This equation represents... On the epipolar line in the second image. When the number of inliers satisfying the above fundamental matrix equation is the largest, i.e. greater than the preset third threshold, the rotation matrix R and translation vector t are decomposed from the fundamental matrix F to construct the pose transformation matrix.

[0093] The essential matrix E is a 3x3 matrix used to describe the epipolar geometry between two calibrated images. It is similar to the fundamental matrix, but for a calibrated camera, the essential matrix contains the camera's intrinsic parameters. For corresponding points in the two images... and They satisfy the following equation:

[0094]

[0095] in, and This is the intrinsic parameter matrix of the camera for two images. This equation also represents the point... and On the epipolar line in the second image. When the number of inliers satisfying the above fundamental matrix equation is the largest, that is, greater than the preset fourth threshold, the rotation matrix R and translation vector t are decomposed from the essential matrix E to construct the pose transformation matrix.

[0096] In 2D-2D matching, the RANSAC (Random Sample Consensus) algorithm is used to estimate the fundamental matrix F or the essential matrix E. The RANSAC algorithm estimates these matrices by randomly sampling matching point pairs and calculates the number of point pairs that satisfy the equation. These point pairs that satisfy the equation are called inliers. The more inliers there are, the more reliable the estimated pose transformation matrix is.

[0097] Step S600: Determine the pose transformation matrix corresponding to the corresponding feature point;

[0098] The pose transformation matrix is ​​calculated on an image-by-image basis, prioritizing 3D-3D matching mode over 2D-3D matching mode, and 2D-3D matching mode over 2D-2D matching mode. The matrix is ​​then evaluated by checking for inlier count thresholds and / or inlier ratio thresholds. If a threshold is not met, the calculation is performed at a lower priority level to determine the corresponding pose transformation matrix. Alternatively, after estimating the pose transformation matrix using the above steps, the inlier count is counted, and the matrix with the highest inlier count (i.e., the most reliable) is selected as the pose transformation matrix.

[0099] In the above steps, for 3D-3D correspondences, i.e., when corresponding 3D feature points exist in two sub-maps, the system employs a rigid body transformation algorithm to accurately calculate the relative pose between the two sub-maps by minimizing the distance between pairs of 3D feature points. This method utilizes direct geometric constraints in 3D space, offering higher accuracy and robustness compared to 2D-3D or 2D-2D matching. It effectively handles situations with inconsistent scales and large viewpoint changes, significantly improving the accuracy of multi-source data fusion and enabling automatic alignment and fusion of multi-source image data without the need for additional hardware.

[0100] Step S700: Based on the pose transformation matrix corresponding to the corresponding feature points, perform preliminary alignment of the image data contained in the similar image pair in the global coordinate system to generate a fused global model.

[0101] The purpose of initial alignment of multiple submaps is to solve for all submaps. Transformation to a selected global coordinate system In step S600 above, a large number of "anchor points" connecting different sub-maps are determined, namely, those successfully matched image pairs and their relative pose transformations. (indicating from) coordinate system to (Coordinate system transformation). Here, we construct a graph optimization problem and minimize the relative pose constraint error to achieve this.

[0102] Let the global coordinate system be... The pose of image i in submap m is (In its local coordinate system), its expression in the global coordinate system is: Similarly, the pose of image j in subgraph n in the global frame is... .

[0103] According to the matching pair Set a constraint: the relative pose calculated through image matching. It should be equal to the relative transformation of these two poses in the global frame:

[0104]

[0105] Here, a least squares problem is constructed to solve for the transformation of all subgraphs to the global system:

[0106] ;

[0107] ;

[0108] Solving this problem yields the transformation matrix that initially aligns all sub-maps to the global coordinate system.

[0109] Step S800: Perform global optimization on the fused global model to complete the reconstruction of the 3D map.

[0110] In the fused global model, a global bundle adjustment optimization problem is constructed using all camera poses and 3D points as optimization variables, and the fused global model is then globally optimized.

[0111] When performing global optimization, cross-scale optimization can be performed on sub-maps from different sources. Specifically, the projection function is: ;

[0112] The objective function is to minimize the reprojection error between all observed pixels and projected pixels:

[0113] ;

[0114] in, It is Mahalanobis distance. For the 3D point in the i-th camera pose The observed 2D pixel coordinates, It is the covariance matrix of the observed value, which is related to the feature point detection accuracy. It is usually assumed to be... , It is a robust kernel function (such as the Huber kernel) used to suppress the effects of outliers; It is the introduced point-to-surface geometric constraint error term. It is the weighting coefficient.

[0115] To achieve cross-scale optimization, observations from different sub-map sources are assigned different weights. These weights are determined based on the sensor working distance *d*, focal length *f*, or image resolution *r* of the sub-map to which the observation belongs. One feasible setting is:

[0116]

[0117] That is, the greater the working distance, the shorter the focal length, and the lower the resolution, the greater the uncertainty of the observation is considered, and the lower its weight in the optimization. The working distance, focal length, or image resolution of the aforementioned sensors can be determined statistically or empirically.

[0118] Furthermore, to minimize the reprojection error between all observed pixels and projected pixels, a scale consistency constraint can be added in this embodiment. To ensure scale consistency, the scale factor between sub-maps is included in the pose estimation and reprojection error calculation. To fully utilize the structured features in the scene, and considering that the sparse point cloud is optimized during sub-map construction, if the sparse point cloud map of the sub-map meets the flatness condition, point-to-surface error will be added to minimize the error of the sparse point cloud between sub-maps.

[0119] First, the sparse point cloud set corresponding to the image data contained in the similar image pair is decomposed, and its covariance matrix is ​​calculated.

[0120] For a sparse point cloud set in a submap Perform principal component analysis (PCA) or SVD decomposition on it and calculate its covariance matrix:

[0121]

[0122] The flatness of the sparse point cloud is determined based on the covariance matrix, and the flatness is defined as follows:

[0123]

[0124] If the flatness is less than or equal to a preset flatness threshold, that is, if , If a preset flatness threshold is set, the point cloud is determined to be a plane, and the point-to-plane error term is then defined. The error term is the sum of the distances from all points in the point cloud to its fitted plane:

[0125]

[0126] The fused global model is globally optimized based on the point-surface error term. This error term will enable the optimized point cloud to better conform to the planar structure and effectively constrain the geometric shape inside the sub-image and the relative pose between sub-images.

[0127] If the flatness is greater than the preset flatness threshold, the point cloud that does not meet the planarity condition will not be subject to this constraint and will degenerate into the standard reprojection error and epipolar geometry constraint.

[0128] Furthermore, sparse nonlinear optimization solvers (such as Ceres Solver, g2o, etc.) can be used for global optimization to obtain the final accurately aligned and geometrically consistent multi-source 3D map reconstruction result. In this embodiment, to improve optimization efficiency and reduce the computational complexity of a single iteration, this application adopts an optimization strategy of alternating fixed control points, such as... Figure 3 As shown, it includes:

[0129] Step S810: Determine the common visual feature points among the image data contained in the similar image pairs to form a set of common visual feature points;

[0130] Commonly observed feature points refer to the same three-dimensional spatial point that can be observed in different images or viewpoints. These points have corresponding two-dimensional projection points in different images. By matching these two-dimensional projection points, the relative pose of the camera and the coordinates of the three-dimensional point can be estimated. This application identifies the commonly observed feature points between image data of similar image pairs and forms a set of all commonly observed feature points.

[0131] Step S820: Randomly select a preset proportion of common-view feature points from the set of common-view feature points as fixed control points;

[0132] Commonly seen feature points (3D points) between submaps are key to connecting different submaps. Here, a binary marker container M and a counter C are defined. For each global iteration, a certain percentage (e.g., n%, 20%) of the points in the common-view feature point set are randomly selected as control points (fixed points) for this iteration. The state of these points is set to fixed in the marker container M. The counter C is incremented by n.

[0133] Step S830: Perform global optimization on the fused global model and reduce the preset ratio until all the common-view feature points are used as fixed control points.

[0134] In each iteration of the optimization problem, all fixed control points have their parameters fixed and are no longer considered optimization variables. Only the camera pose and 3D points that are not fixed are optimized. This significantly reduces the number of variables in each iteration and improves computation speed.

[0135] Repeat the above process until the counter C reaches or exceeds 100% (i.e., all common feature points have been fixed at least once), or the objective function converges to a predetermined threshold. This alternating approach ensures that all observation information is ultimately used to constrain the global model, while also significantly improving optimization efficiency.

[0136] As can be seen from the above, the embodiments of this application improve the accuracy and robustness of feature matching by combining global features and local features, and introduce point-plane geometric constraints and common feature point alternation optimization strategies to ensure the global consistency of the model after multi-source data fusion.

[0137] In summary, the 3D map reconstruction method and device proposed in this application determine corresponding feature points by performing local feature matching on similar image pairs. For 3D-3D correspondences, i.e., when corresponding 3D feature points exist in two sub-maps, the system employs a rigid body transformation algorithm to accurately calculate the relative pose between the two sub-maps by minimizing the distance between the 3D feature point pairs. This approach does not rely on external positioning devices such as GNSS / RTK, reducing costs and deployment difficulty. It boasts a high degree of automation, automatically completing sub-map alignment and fusion based on image features throughout the entire process, eliminating the need for manual point selection. Furthermore, this application also employs a combination of global and local features to improve the accuracy and robustness of feature matching, and introduces point-plane geometric constraints and a common-view feature point alternating optimization strategy to ensure the global consistency of the model after multi-source data fusion. It is adaptable to imaging devices of different brands, models, and resolutions, exhibiting strong compatibility.

[0138] This application also provides a computer-readable storage medium storing executable instructions. When the executable instructions are run on a 3D map reconstruction device, the 3D map reconstruction device performs the 3D map reconstruction method provided in any of the above embodiments.

[0139] This application also provides a three-dimensional map reconstruction program, which is used to execute the three-dimensional map reconstruction method provided in the above embodiments.

[0140] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0141] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0142] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of this application, various features of the embodiments of this application are sometimes grouped together into a single embodiment, figure, or description thereof.

[0143] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying abstract and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying abstract and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0144] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can design alternative embodiments without departing from the scope. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.

Claims

1. A method for reconstructing a three-dimensional map, characterized in that, include: Multiple image data are acquired, and features are extracted from the multiple image data. Multiple sub-maps are generated by local sparse reconstruction of multiple image data with overlapping viewpoints. Global feature extraction is performed on multiple sub-maps to generate a global feature database; Similar image pairs are determined from the global feature database based on the similarity of each of the sub-maps; Local feature matching is performed on the similar image pairs to determine the corresponding feature points of the image data contained in the similar image pairs; If the corresponding feature point has a corresponding 3D feature point in the respective sub-map coordinate system of the image data, then a rigid body transformation is performed on the 3D feature point to make the distance between the 3D feature points less than a preset first threshold, and the pose transformation matrix corresponding to the corresponding feature point is determined. If the corresponding feature point has a 3D feature point and a corresponding 2D feature point in the respective sub-map coordinate system of the image data, then the reprojection error of the 3D feature point on the 2D feature point is determined, such that the reprojection error is less than a preset second threshold, and the pose transformation matrix corresponding to the corresponding feature point is determined. Based on the pose transformation matrix corresponding to the corresponding feature points, the image data contained in the similar image pairs are initially aligned in the global coordinate system to generate a fused global model. The fused global model is then globally optimized to complete the reconstruction of the 3D map.

2. The method according to claim 1, characterized in that, The process of acquiring multiple image data, extracting features from the multiple image data, and performing local sparse reconstruction of the multiple image data with overlapping viewpoints to generate multiple sub-maps includes: The image data is preprocessed; Feature points are extracted from the preprocessed image data, and a local sparse point cloud is constructed based on the feature points; The pose data corresponding to the image data is obtained by local beam adjustment based on the local sparse point cloud. Based on the pose data, multiple sub-maps are generated by locally sparse reconstruction of multiple image data with overlapping viewpoints.

3. The method according to claim 1, characterized in that, The global feature database includes global feature descriptors for the image data contained in each sub-map; The step of determining similar image pairs from the global feature database based on the similarity of each of the sub-maps includes: Determine the global feature descriptor of the image data to be queried; Determine the cosine similarity between the global feature descriptor of the image data to be queried and the global feature descriptor in the global feature database; The similar image pairs contained in the sub-map are determined based on the cosine similarity.

4. The method according to claim 3, characterized in that, The step of performing local feature matching on the similar image pairs to determine the corresponding feature points of the image data contained in the similar image pairs includes: Extract local feature points and global feature descriptors from the image data contained in the similar image pairs; Based on the local feature points and global feature descriptors, a nearest neighbor search algorithm is used to perform local feature matching on the image data contained in the similar image pairs; Determine the corresponding feature points in the image data contained in the similar image pair.

5. The method according to claim 1, characterized in that, The method further includes: If the corresponding feature point has a corresponding 2D feature point in the respective sub-map coordinate system of the image data, then the fundamental matrix of the image data contained in the similar image pair is determined, or the essential matrix of the image data contained in the similar image pair is determined, so that the number of feature points satisfying the fundamental matrix equation is greater than a preset third threshold, or the number of feature points satisfying the essential matrix equation is greater than a preset fourth threshold, and the pose transformation matrix corresponding to the corresponding feature point is determined.

6. The method according to claim 1, characterized in that, The step of globally optimizing the fused global model to reconstruct the 3D map includes: The fused global model is globally optimized using the global bundle adjustment method.

7. The method according to claim 6, characterized in that, The global optimization of the fused global model using the global bundle adjustment method includes: The sparse point cloud set corresponding to the image data contained in the similar image pair is decomposed, and its covariance matrix is ​​calculated. The flatness of the sparse point cloud is determined based on the covariance matrix. If the flatness is less than a preset flatness threshold, then a point-to-surface error term is determined, and the fused global model is globally optimized based on the point-to-surface error term.

8. The method according to claim 6, characterized in that, The global optimization of the fused global model using the global bundle adjustment method includes: Determine the common visual feature points among the image data contained in the similar image pairs to form a set of common visual feature points; Randomly select a preset proportion of common-view feature points from the set of common-view feature points as fixed control points; The fusion global model is globally optimized, and the preset ratio is reduced until all the common-view feature points are used as fixed control points.

9. A three-dimensional map reconstruction device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one program that causes the processor to perform the operation of the three-dimensional map reconstruction method as described in any one of claims 1-8.

Images