Three-dimensional reconstruction method, apparatus, computing device, and storage medium

Abstract

The application provides a three-dimensional reconstruction method, a three-dimensional reconstruction device, a computing device and a computer readable storage medium. The method comprises: acquiring scene data collected by a collection device for an original scene; dividing the scene data into multiple sub-classes, each of the multiple sub-classes comprising internal nodes exclusive to the sub-class and common nodes shared with adjacent sub-classes, and the sub-classes being a collection of images, voxels, point clouds and / or meshes; determining a relative geometric transformation relationship between local coordinate systems of two adjacent sub-classes according to the common nodes between the two adjacent sub-classes; determining an absolute geometric transformation relationship of the local coordinate systems of the sub-classes relative to a global coordinate system according to the relative geometric transformation relationship of the multiple sub-classes; and constructing a three-dimensional model of the original scene according to the absolute geometric transformation relationship of the multiple sub-classes. According to the application, the motion trajectory of the collection device can be accurately restored, and an accurate three-dimensional model can be formed.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a three-dimensional reconstruction method, a three-dimensional reconstruction device, a computing device, and a computer-readable storage medium. Background Technology

[0002] 3D reconstruction technology is known in this field. Existing 3D reconstruction technologies often struggle to achieve a balance between cost, speed, and accuracy, often resulting in either excessively high costs, slow reconstruction speeds, or insufficient accuracy in the reconstructed 3D models. Some 3D reconstruction techniques (such as regional offline reconstruction) attempt to address these issues. However, when aggregating subclasses, such techniques often suffer from misalignment of the acquisition device trajectories across different subclasses, leading to reconstructed 3D models with relatively low accuracy. Summary of the Invention

[0003] The purpose of this application is to provide a three-dimensional reconstruction method, a three-dimensional reconstruction device, a computing device, and a computer-readable storage medium that can balance reconstruction cost, speed, and accuracy, and can achieve good alignment of the acquisition device trajectory between various subclasses.

[0004] In one aspect, this application provides a 3D reconstruction method, comprising: acquiring scene data collected by an acquisition device for an original scene; dividing the scene data into multiple subclasses, each subclass including internal nodes specific to the subclass and common nodes shared with adjacent subclasses, the subclass being a collection of images, voxels, point clouds, and / or meshes; determining the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses based on the common nodes between every two adjacent subclasses; determining the absolute geometric transformation relationship between the local coordinate system of the subclasses and the global coordinate system based on the relative geometric transformation relationship of the multiple subclasses; and constructing a 3D model of the original scene based on the absolute geometric transformation relationship of the multiple subclasses.

[0005] According to this application, the absolute pose of each subclass can be solved and optimized by using the relative geometric transformation relationship between subclasses. This avoids optimizing the two strongly coupled variables of relative geometric transformation relationship and camera pose together, thereby achieving higher reconstruction efficiency and better alignment of camera trajectories.

[0006] According to a particular embodiment of this application, scene data is divided into multiple subclasses, including: dividing scene data into multiple subclasses using a graph cut algorithm.

[0007] According to this embodiment, the graph cut algorithm divides the data into two parts, minimizing the connections between these two parts while maximizing the connections within each part. By classifying the scene data using the graph cut algorithm, multiple subclasses with high internal data correlation can be obtained, while ensuring that there are few connecting nodes between subclasses, which is beneficial for achieving accurate and efficient segmentation.

[0008] According to a particular embodiment of this application, two adjacent subclasses share multiple common nodes. Determining the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses based on the common nodes between each pair of adjacent subclasses includes: determining multiple relative geometric transformation relationships constituting a set of relative geometric transformation relationships between the local coordinate systems of two adjacent subclasses based on the multiple common nodes between each pair of adjacent subclasses. Determining the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system based on the relative geometric transformation relationships of the multiple subclasses includes: determining the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system based on the multiple sets of relative geometric transformation relationships of the multiple subclasses.

[0009] According to this embodiment, since there are multiple common nodes between subclasses, there are various possible transformation relationships. By taking all these possible transformation relationships into account and optimizing them together, it is beneficial to accurately determine the real transformation relationship between subclasses, so that the camera motion trajectories between subclasses can be better connected, thereby constructing an accurate 3D model.

[0010] According to a particular embodiment of this application, determining the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system based on multiple sets of relative geometric transformation relationships of multiple subclasses includes: calculating the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system using an L1 loss function based on multiple sets of relative geometric transformation relationships of multiple subclasses.

[0011] According to this embodiment, optimizing multiple sets of relative geometric transformation relationships among all subclasses using the L1 loss function has been proven to achieve simple and efficient calculations. By setting the optimization objective of minimizing the L1 loss, the entire optimization process becomes fast and controllable, achieving the goal of accurately determining the pose of the subclasses.

[0012] According to a particular embodiment of this application, a three-dimensional model of the original scene is constructed based on the absolute geometric transformation relationships of multiple subclasses, including: determining the acquisition device pose and three-dimensional points of the subclasses in the global coordinate system based on the absolute geometric transformation relationships of the subclasses; and constructing a three-dimensional model of the original scene based on the acquisition device pose and three-dimensional points of the multiple subclasses.

[0013] According to this embodiment, the pose of the acquisition device or camera is calculated by using absolute geometric transformation relationships, and then the 3D points are calculated. This has been shown to save computing resources and avoid the problem of difficulty in convergence caused by optimizing the two variables of geometric transformation relationships and camera pose that are coupled together.

[0014] According to a particular embodiment of this application, the 3D reconstruction method further includes: obtaining the model region to be updated drawn by the user on the 3D model of the original scene; deleting the model region to be updated; merging the scene data of the remaining model region into a pre-update subclass; dividing the scene data of the updated scene into multiple update subclasses; selecting common nodes between the pre-update subclass and the update subclass on the boundary of the remaining model region; merging the pre-update subclass and the multiple update subclasses into multiple post-update subclasses; and constructing a 3D model of the updated scene based on the multiple post-update subclasses.

[0015] This embodiment provides a method for updating a 3D model. Since the 3D model of the original scene is obtained through the aforementioned 3D reconstruction method, updating the 3D model can be achieved more simply and quickly through the steps of this embodiment. Specifically, the user only needs to select the model area to be updated or replaced, and select the common nodes between the subclass of the updated scene and the subclass of the original scene to update the model. This embodiment can support heterogeneous data and ensure that the updated scene blends well with the original scene.

[0016] According to a particular embodiment of this application, the 3D reconstruction method further includes: obtaining the boundary between the 3D model of the extended scene and the 3D model of the original scene; merging the scene data of the original scene into a pre-extension subclass; dividing the scene data of the extended scene into multiple extended subclasses; selecting common nodes between the pre-extension subclasses and the extended subclasses on the boundary; merging the pre-extension subclasses and the multiple extended subclasses into multiple post-extension subclasses; and constructing a 3D model of the extended scene based on the multiple post-extension subclasses.

[0017] This embodiment provides a method for expanding a 3D model. This expansion method is based on the aforementioned 3D reconstruction method, thus enabling simpler and faster scene expansion. Specifically, the user only needs to select the common nodes between the subclasses of the expanded scene and the subclasses of the original scene to expand the 3D model. This expansion method supports heterogeneous data and ensures that the expanded scene blends well with the original scene.

[0018] According to a particular embodiment of this application, the relative geometric transformation relationship includes the relative scale transformation relationship, the relative rotation transformation relationship, and the relative translation transformation relationship, and the absolute geometric transformation relationship includes the absolute scale transformation relationship, the absolute rotation transformation relationship, and the absolute translation transformation relationship.

[0019] According to this embodiment, by defining geometric transformations as scaling transformations, rotation transformations, and translation transformations, the meaning of geometric transformations can be further clarified, the operation of geometric transformations can be simplified, and the implementation of geometric changes can be made simpler and more efficient.

[0020] According to a particular embodiment of this application, the data acquisition device includes one or more of the following: inertial sensor, lidar, ultrasonic radar, millimeter-wave radar, visible light camera, and infrared camera.

[0021] According to this embodiment, users can use one or more acquisition devices to collect data from the scene. Regardless of the combination method used, the three-dimensional reconstruction method of this application can be used to construct a three-dimensional model, making the application scenarios of this application more extensive.

[0022] On the other hand, this application provides a three-dimensional reconstruction apparatus, comprising: a first acquisition module for acquiring scene data collected by an acquisition device for an original scene; a first segmentation module for dividing the scene data into multiple subclasses, each subclass including internal nodes specific to the subclass and common nodes shared with adjacent subclasses; a first determination module for determining the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses based on the common nodes between every two adjacent subclasses; a second determination module for determining the absolute geometric transformation relationship between the local coordinate system of a subclass and the global coordinate system based on the relative geometric transformation relationship of the multiple subclasses; and a first construction module for constructing a three-dimensional model of the original scene based on the absolute geometric transformation relationship of the multiple subclasses.

[0023] According to a particular embodiment of this application, the first segmentation module is further configured to: divide the scene data into multiple subclasses using a graph cut algorithm.

[0024] According to a particular embodiment of this application, two adjacent subclasses share multiple common nodes. The first determining module is further configured to: determine multiple relative geometric transformation relationships between the local coordinate systems of two adjacent subclasses, constituting a set of relative geometric transformation relationships, based on the multiple common nodes between each pair of adjacent subclasses. The second determining module is further configured to: determine the absolute geometric transformation relationship between the local coordinate system of a subclass and the global coordinate system, based on the multiple sets of relative geometric transformation relationships of the multiple subclasses.

[0025] According to a particular embodiment of this application, the second determining module is further configured to: calculate the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system using an L1 loss function based on multiple sets of relative geometric transformation relationships of multiple subclasses.

[0026] According to a particular embodiment of this application, the first construction module is further configured to: determine the acquisition device pose and 3D points of the subclass in the global coordinate system based on the absolute geometric transformation relationship of the subclass; and construct a 3D model of the original scene based on the acquisition device poses and 3D points of multiple subclasses.

[0027] According to a particular embodiment of this application, the 3D reconstruction device further includes: a second acquisition module for acquiring the model region to be updated drawn by the user on the 3D model of the original scene; a deletion module for deleting the model region to be updated; a first merging module for merging the scene data of the remaining model regions into a pre-update subclass; a second segmentation module for dividing the scene data of the updated scene into multiple update subclasses; a first selection module for selecting common nodes between the pre-update subclasses and the update subclasses on the boundaries of the remaining model regions; a second merging module for merging the pre-update subclasses and multiple update subclasses into multiple post-update subclasses; and a second construction module for constructing a 3D model of the updated scene based on the multiple post-update subclasses.

[0028] According to a particular embodiment of this application, the 3D reconstruction apparatus further includes: a third acquisition module for acquiring the boundary between the 3D model of the extended scene and the 3D model of the original scene; a third merging module for merging the scene data of the original scene into a pre-extension subclass; a third segmentation module for dividing the scene data of the extended scene into multiple extended subclasses; a second selection module for selecting common nodes between the pre-extension subclasses and the extended subclasses on the boundary; a fourth merging module for merging the pre-extension subclasses and multiple extended subclasses into multiple post-extension subclasses; and a third construction module for constructing a 3D model of the extended scene based on the multiple post-extension subclasses.

[0029] On the other hand, this application provides a computing device including a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the above-described three-dimensional reconstruction method.

[0030] On the other hand, this application provides a computer-readable storage medium storing a computer program for performing the above-described three-dimensional reconstruction method.

[0031] On the other hand, this application provides a computer program product, including program code, which, when a computer runs the computer program product, enables the computer to implement the above-described three-dimensional reconstruction method.

[0032] Any of the devices, computer-readable storage media, or computer program products provided above are used to perform the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding solutions in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0033] The specific embodiments of this application are described in detail below with reference to the accompanying drawings, wherein:

[0034] Figure 1 This application shows a schematic diagram of the logical architecture of various embodiments;

[0035] Figure 2 A flowchart illustrating a three-dimensional reconstruction method according to an embodiment of this application is shown;

[0036] Figure 3 Showing according to Figure 2 The reconstruction effect diagram of the three-dimensional reconstruction method in the embodiment;

[0037] Figure 4 A flowchart illustrating a three-dimensional reconstruction method according to another embodiment of this application is shown;

[0038] Figure 5 A flowchart illustrating a three-dimensional reconstruction method according to another embodiment of this application is shown;

[0039] Figure 6 A schematic diagram of the structure of a three-dimensional reconstruction apparatus according to an embodiment of this application is shown;

[0040] Figure 7 A schematic diagram of the structure of a three-dimensional reconstruction apparatus according to another embodiment of this application is shown;

[0041] Figure 8 A schematic diagram of the structure of a three-dimensional reconstruction apparatus according to another embodiment of this application is shown;

[0042] Figure 9 A schematic diagram of the structure of a computing device according to an embodiment of this application is shown. Detailed Implementation

[0043] To enable those skilled in the art to more clearly understand the concepts and ideas of this application, the application is described in detail below with reference to specific embodiments. It should be understood that the embodiments given herein are only a part of all possible embodiments of this application. After reading the specification of this application, those skilled in the art are capable of making improvements, modifications, or substitutions to parts or the entirety of the following embodiments, and such improvements, modifications, or substitutions are also included within the scope of protection claimed in this application.

[0044] In this document, the terms "one," "an," and other similar words are not intended to indicate that only one of the described things exists, but rather that the description refers only to one of the described things, which may have one or more. In this document, the terms "comprising," "including," and other similar words are intended to indicate a logical relationship, not a spatial relationship. For example, "A includes B" means that logically B belongs to A, not that spatially B is located inside A. Furthermore, the meanings of the terms "comprising," "including," and other similar words should be considered open-ended, not closed-ended. For example, "A includes B" means that B belongs to A, but B does not necessarily constitute all of A; A may also include other elements such as C, D, and E.

[0045] In this document, the terms "first," "second," and other similar terms are not intended to imply any order, quantity, or importance, but are merely used to distinguish different elements. In this document, the terms "embodiment," "this embodiment," "an embodiment," or "an example" do not indicate that the description applies only to one specific embodiment, but rather that such description may also be applicable to one or more other embodiments. Those skilled in the art will understand that any description made herein with respect to one embodiment can be substituted, combined, or otherwise combined with the descriptions in one or more other embodiments, and the new embodiments resulting from such substitutions, combinations, or other combinations are readily conceived by those skilled in the art and fall within the scope of protection of this application.

[0046] In the embodiments of this application, 3D reconstruction can refer to establishing a mathematical model suitable for computer representation and processing of a 3D object. It is the foundation for processing, manipulating, and analyzing the properties of the object in a computer environment, and a key technology for establishing virtual reality that expresses the objective world in a computer. For example, 3D reconstruction can refer to the mathematical process and computer technology of recovering the 3D information (shape, etc.) of an object using 2D projection or images. In computer vision, 3D reconstruction refers to the process of reconstructing 3D information from single-view or multi-view images. Because the information in a single view is incomplete, 3D reconstruction requires the use of empirical knowledge. Multi-view 3D reconstruction (similar to human binocular positioning) is relatively easier; the method involves first calibrating the camera, i.e., calculating the relationship between the camera's image coordinate system and the world coordinate system. Then, 3D information is reconstructed using information from multiple 2D images.

[0047] 3D reconstruction is a hot topic in computer vision and computer graphics, and 3D scene reconstruction is one of the most critical technologies. 3D scene maps (primarily referring to a 3D representation of a scene including point clouds, visual feature points, and their descriptors, possessing location information and mainly used for positioning and navigation) and 3D models (usually referring to a 3D representation of a scene including surfaces, textures, and even materials, consistent with the appearance of the real scene, mainly used for visualization and virtual-real overlay) have wide applications and high practical value. Specifically, in the extended reality (XR) field, applications such as augmented reality (AR) inspection, mixed reality (MR) city navigation, MR smart business districts, holographic conferencing, and remote assistance all require 3D scene maps and models as a foundation. In the robotics field, the localization, navigation, and route planning of mobile robots all rely on 3D scene maps, exhibiting a strong dependence on the ability to construct large-scale, high-precision 3D scene maps. With the rise of the metaverse concept, applications such as virtual shopping, autonomous driving simulation, and open-world games all demand efficient large-scale 3D scene reconstruction capabilities. Smart city scenarios also require modeling of large-scale scenarios such as parks and cities, which can serve as a visualization base for digital twins and be used for city management, planning, measurement, simulation, etc.

[0048] 3D scene reconstruction typically employs devices such as 3D lasers, SLR cameras, and panoramic cameras to acquire raw scene data. Other devices include inertial measurement units (IMUs), depth cameras, and mobile phones. Often, multiple sensors are used simultaneously to collect various types of data for 3D scene reconstruction. Using the acquired data as input, 3D scene reconstruction typically utilizes techniques such as simultaneous localization and mapping (SLAM), vision-inertial SLAM, and structure from motion (SFM) to construct 3D maps and models.

[0049] 3D reconstruction based on various SLAM technologies falls under the category of online reconstruction or online mapping techniques. While offering high reconstruction efficiency, allowing for simultaneous scanning (data acquisition) and reconstruction, its accuracy is often limited, and it can only be used with temporally continuous data (such as video or continuous laser point cloud frames). Furthermore, it is susceptible to factors like rapid motion. Laser SLAM, benefiting from hardware advantages, can guarantee high geometric accuracy, but its equipment is expensive, lacks portability, and is difficult to scale up. Additionally, lasers themselves are not ideal for constructing visual feature-based 3D maps, requiring additional camera equipment and calibration. SFM-based 3D reconstruction is an offline reconstruction or offline mapping technique. Its computational load is high, making real-time or near-real-time reconstruction impossible, requiring data acquisition before reconstruction. However, it offers high reconstruction accuracy and can be used with unordered image data. SFM primarily uses images as input, thus naturally constructing a visual feature-based 3D map while building a point cloud. It also has low equipment requirements, allowing for the use of portable panoramic cameras, SLR cameras, and mobile phones. Where feasible, laser point cloud data can also be used to enhance geometric accuracy. In summary, some 3D scene reconstruction methods in this field either rely on high-cost, heavy hardware, have difficulty guaranteeing accuracy, or have significant shortcomings in reconstruction efficiency, thus often encountering many difficulties and problems in practical applications.

[0050] After reconstructing the 3D map and model of the scene, since the real world is constantly changing and the scene may continue to expand, there is also the problem of scene updating (partial replacement and expansion). That is, after the scene changes, how to locally replace the 3D map and model in this field, and when it is necessary to expand the scene, how to merge the new 3D map and model into the existing map and model, and after updating the 3D scene, ensure that the replaced and added parts maintain good consistency with the original 3D scene, and that the spatial 3D points do not show obvious breaks, misalignments, ghosting, etc.

[0051] To update existing 3D scenes, the most direct approach is to reconstruct the 3D scene after acquiring new data. While this method maintains good consistency, it is computationally intensive and requires the acquired data to be homogeneous—that is, collected by the same type of equipment and containing the same data. Another approach is to reconstruct the scene based on newly acquired data and then replace a portion of the existing 3D scene with the new scene, or stitch it together with the existing scene. This approach can support heterogeneous data, but it struggles to guarantee seamless consistency in the stitched 3D scenes. Manual operation is particularly prone to errors at the update edges, while automated methods also suffer from significant shortcomings in efficiency and accuracy.

[0052] Based on the above analysis, some technical solutions in this field have significant shortcomings in one or more aspects such as ease of use, efficiency, and accuracy in 3D scene reconstruction, and are difficult to effectively support the updating and expansion of 3D scenes, especially under heterogeneous data conditions, where efficiency and accuracy are relatively limited.

[0053] Some 3D scene reconstruction techniques in this field can be divided into two main categories: online reconstruction and offline reconstruction. Offline reconstruction schemes can be further divided into overall reconstruction and region-based reconstruction. In addition, some techniques specifically address the updating of 3D scenes.

[0054] Online reconstruction mainly refers to scene reconstruction using various SLAM methods. While using sensors to capture scene data, reconstruction is performed simultaneously. In robotics, online reconstruction is the most common scene reconstruction method. Robots are equipped with one or more sensors to simultaneously scan environmental data and build 3D maps. A 3D map is generated in real time during robot operation, typically using laser equipment to construct a 3D point cloud map of the scene through laser SLAM.

[0055] These methods perform feature matching on adjacent frames within a time window, employing techniques such as iterative closest point (ICP), Kalman filtering, and graph optimization to calculate and optimize pose (camera pose / image pose, which can refer to the 3D position and 3D rotation angle of the image, i.e., the transformation of visible 3D points on the image to the world coordinate system) for registration, obtaining a local 3D scene, and then gradually generating the entire 3D scene. To reduce equipment costs and ensure reconstruction accuracy, some techniques in this field, based on single-line LiDAR, uniformly extract features such as curvature by region and remove problematic feature points.

[0056] Offline reconstruction primarily refers to scene reconstruction based on the SFM method, which performs reconstruction after data acquisition. This type of approach typically uses an image set as input, extracts feature points and corresponding feature descriptors from the images, performs image matching to obtain matching relationships between images, and then further employs algorithms such as n-point perspective (PNP), triangulation, bundle adjustment, and BA to estimate camera pose and 3D point coordinates of the scene. Pose estimation is the core issue addressed by SFM, as it forms the basis for calculating the (sparse or dense) 3D model of the scene. Typical SFM methods include two main categories: incremental SFM and global SFM.

[0057] Incremental SFM methods first select a pair of initial images for reconstruction, and then progressively select new images to register with the existing model. After registering one or more images, global optimization is performed through BA. Representative methods include Photo Tourism, Visual SFM, and Adaptive Structure from Motion (ASFM). Because incremental reconstruction methods require registration of each image individually, and global BA needs to be performed after each registered image to reduce error accumulation, the efficiency is generally low, and drift is still difficult to avoid.

[0058] Global SFM solves the global pose for all cameras in one go using motion averaging, followed by a global BA. Representative methods include 1-dimensional structure from motion (1DSFM) and least unsquared deviations (LUD). Motion averaging is a key technique in global SFM, encompassing rotational averaging and translational averaging. Rotational averaging methods include efficient and robust large-scale rotation averaging (ERLS-RA). Compared to incremental SFM, global SFM considers all spatially adjacent relative constraints in the global domain, avoiding drift caused by error accumulation. Representative methods include global structure-from-motion by similarity averaging. The Global Averaging Method (GSA-SFM) solves for the absolute pose of each star map (and consequently each camera within the star map) in the global coordinate system. Specifically, it constructs a star map from locally adjacent cameras, and the centers of spatially adjacent star maps, such as i and j, are simultaneously contained within star maps i and j. It calculates the relative scale, relative rotation, and relative translation between the centers (represented as i and j) of adjacent star maps in the coordinate systems of the two star maps i and j, based on the values ​​of these values. Then, it summarizes the relative constraints between all adjacent star maps globally into a system of equations. By minimizing the L1 loss, it finds the global optimal solution that satisfies all relative constraints globally (between spatially adjacent star maps), obtaining the absolute scale and absolute pose of the origin of each star map in the global coordinate system, and thus the pose of each camera in the global coordinate system.

[0059] Most surface-level motion mapping (SFM) methods are geometry-based. While deep learning-based methods also exist, their applicability is limited and they remain in the exploratory stage. Deep learning-based methods can be broadly categorized into two types: one uses networks to predict depth maps and camera poses separately or jointly, minimizing photometric consistency and introducing geometric constraints to assist in optimization, directly predicting depth maps and camera poses, with BA-Net as a representative example; the other directly predicts voxels, point clouds, or meshes, such as the DeepSymmetric Distance Function (DeepSDF), OctNet (Octtree Network), DeformNet (Deformation Network), and 3D-R2N2 (3D Recursive Reconstruction Neural Network). This latter method is typically only suitable for 3D reconstruction of small scenes and cannot be used for large-scale scenes.

[0060] While offline reconstruction methods offer high accuracy, their efficiency is low and reconstruction time is long for large-scale scenes due to the large amount of data. To address this issue, some methods employ a region-based reconstruction approach. The main idea is to first cluster the image, dividing a scene into different local scenes, and then reconstruct each local scene separately (usually using incremental SFM). Finally, these local 3D scenes are fused (usually using a moving average algorithm) to obtain the final complete 3D scene. Representative methods include graph cut-based methods and spanning tree-based methods. This approach combines the advantages of incremental and global SFM to some extent, improving reconstruction speed while maintaining robustness.

[0061] A representative method for offline reconstruction of regions includes the efficient large-scale motion inference structure from motion using graph partitioning (DivConq-SFM) and the very large-scale global SFM by distributed motion averaging (VLSG-SFM). VLSG-SFM divides the entire scene graph to obtain multiple subgraphs corresponding to local scenes. It then performs distributed local reconstruction on each subgraph (estimating the pose of each camera within the subgraph in the local subgraph coordinate system). Next, it alternately performs global motion averaging on the inter-nodes (common nodes, shared boundary nodes) between subgraphs and local motion averaging on the intra-nodes (internal nodes) within the subgraphs. Simultaneously, it solves for the scale transformation from the subgraph coordinate system to the global coordinate system, the similarity transformation, and the global pose of the inter-nodes. Its main steps include:

[0062] 1) Extract feature points from all images in the entire scene, perform image matching based on feature point matching, and construct an edge graph (EG). An edge graph is a graph constructed based on the matching relationship of image pairs as edges according to the image matching results.

[0063] 2) Use the graph cut algorithm to divide the entire EG into multiple subgraphs.

[0064] 3) Perform local reconstruction of each sub-graph in a distributed manner to estimate the local pose of all cameras within each subclass.

[0065] 4) Using the DivConq-SFM algorithm, calculate the scale transformation and similarity transformation from each subclass coordinate system to the global coordinate system, and aggregate the subclasses into the global domain as the initial values ​​for motion averaging (MA) loss optimization in step 5.

[0066] 5) For intra-intra and intra-inter pairs of spatial connectivity within local subclasses, construct local MA (Local Rotation Averaging, LRA) and local Translation Averaging, LTA) loss functions. For inter-inter pairs shared between neighboring subclasses in the global space, and inter-intra pairs within a subclass, construct global MA (Global Rotation Averaging, GRA) and global Translation Averaging, GTA) loss functions. Alternately perform LRA, LTA, GRA, and GTA optimizations to optimize the global pose of the inter-camera, and the scale and similarity transformations from the local subclass coordinate system to the global coordinate system.

[0067] In practice, due to changes in the real-world scene or its scope, it is often necessary to update the reconstructed 3D scene, including replacing parts or expanding the 3D scene. The most direct way to update a 3D scene is to reconstruct it again after acquiring new data. While this method can maintain the consistency of the 3D scene relatively well, it involves a large amount of computation and requires that the data acquired before and after the reconstruction be homogeneous, i.e., both acquired by the same type of equipment and containing the same type of data.

[0068] Another approach involves registering and fusing newly acquired data with existing 3D scenes. This can be achieved by first determining the local area to be updated through methods such as setting an image frame age threshold, calculating point cloud differences, or manual selection. Then, point cloud registration or image registration is performed, and methods like rollback mapping, downsampling merging, and octree storage are used to fuse the old and new 3D scenes, thus updating the 3D scene. This type of solution directly fuses data and can only handle homogeneous data.

[0069] Another approach is to reconstruct the existing 3D scene based on newly acquired data, and then replace part of the existing 3D scene with the new 3D scene or stitch it together with the existing 3D scene. This approach can theoretically support heterogeneous data, but it lacks a relevant design. In addition, it is difficult to guarantee the consistency of seamless stitching of 3D scenes. Manual operation is particularly prone to deviations at the update edges, while automatic methods also have obvious shortcomings in terms of efficiency and accuracy.

[0070] The disadvantages of online reconstruction schemes are as follows: 1) LiDAR-based SLAM schemes are typically used for real-time robot localization and mapping, or for MR spatial mapping and scene model building when laser equipment is available. They rely on dedicated equipment, are costly, and lack portability, thus limiting their large-scale adoption. Furthermore, lasers are not ideal for building maps based on visual features, requiring additional cameras and calibration. 2) Visual SLAM schemes (including visual-inertial SLAM) prioritize online operation and real-time performance, employing lightweight features and optimization methods, making it difficult to guarantee reconstruction accuracy. This limitation restricts their application in 3D scene updates. Additionally, online reconstruction schemes can only be used with temporally continuous data, such as video or continuous laser point cloud frames, thus limiting their applicability.

[0071] The disadvantages of holistic offline reconstruction schemes are as follows: 1) Incremental SFM-based schemes are sensitive to the selection of initial image pairs. Iterative image registration is prone to error accumulation leading to drift, requiring repeated BA execution. The algorithm has a large computational load and long reconstruction time. For example, the open-source benchmark software COLMAP takes more than two weeks to complete the reconstruction of 6000 images. 2) Global SFM-based schemes are easily affected by matching outliers. Although computationally efficient, accuracy is poor. Furthermore, when updating the 3D scene, all data needs to be re-collected and global optimized from scratch, which is cumbersome and lacks agility. 3) Deep learning-based schemes are currently in the exploratory stage. They mainly use deep networks for depth and pose estimation, usually relying on large-scale labeled data. Methods such as direct voxel prediction are only suitable for small-scale indoor scenes. When the scene needs to be expanded, due to the limited generalization of deep networks, it is often necessary to collect data from the expanded area and retrain the deep network.

[0072] The disadvantages of the regional offline reconstruction scheme are: relatively speaking, this type of scheme balances reconstruction speed and accuracy, but these methods mainly support image-based reconstruction. The optimization methods used have the problems of being sensitive to the accuracy of scale initialization, strong coupling between the two types of variables in the optimization solution, and do not support heterogeneous data, nor do they have a scheme for updating 3D scenes.

[0073] For example, consider the representative method VLSG-SFM, which iteratively optimizes local and global motion averaging at the camera level. This optimization involves a large number of variables, and it also optimizes two interdependent variables—camera pose and similarity transformations between subclasses—into a single loss function. This strong coupling of variables during optimization leads to convergence difficulties, constituting an ill-conditioned problem. Therefore, reconstruction efficiency and accuracy cannot be guaranteed. Specifically:

[0074] 1) When optimizing the global loss function constructed by merging subgraphs, the true value in the translation loss function depends on the relative scales between subclasses and the relative translations between cameras within subclasses calculated during initialization. Because the relative scales between subclasses calculated during initialization are inaccurate (often some subclasses have significantly deviated from the true value), and because `inter` may belong to more than two subclasses, the selection of the relative translation from any subclass for calculating the true translation in the loss will result in bias. The relative translations between cameras within subclasses may also be inaccurate. This leads to a highly inaccurate true value in the translation loss, making it difficult to correctly guide the optimization of the global loss function for aggregating subclasses. This results in camera trajectories being difficult to align after aggregating subclasses, causing breaks and ghosting issues with 3D points.

[0075] 2) The global motion average loss function includes two types of variables that influence each other: camera pose and similarity transformation between subclasses. These two types of variables are strongly coupled together for optimization. Because the initialization process is inaccurate for both types of variables, and the optimization process does not have a clear and correct optimization direction for the two types of variables, problems such as slow optimization, difficulty in convergence, and getting trapped in local extrema occur, resulting in reconstruction failure.

[0076] The drawbacks of some 3D scene update schemes in this field are as follows: The three schemes mentioned above are all general reconstruction methods, lacking specific consideration and design for 3D scene updates. Therefore, they lack an update mechanism and are not friendly to 3D scene updates. Schemes that directly introduce newly acquired data to reconstruct the entire scene for updates are inefficient and have poor scalability; furthermore, the reliability of the reconstruction decreases as the scene scales up. Schemes that gradually integrate newly acquired data into the existing scene through matching can improve efficiency to some extent, but still require homogeneous data, thus limiting their applicability. Schemes that use new data to reconstruct a local 3D scene separately require replacing or stitching the local 3D scene with the existing 3D scene. Some schemes in this field lack effective overall optimization methods, making it difficult to guarantee the consistency of seamless 3D scene stitching. Manual operation is particularly prone to deviations at the update edges, while automatic optimization using methods like VLSG-SFM also suffers from significant shortcomings in efficiency and accuracy.

[0077] In large-scale scenarios, some 3D scene reconstruction solutions in this field struggle to achieve a balance between reconstruction cost, speed, and accuracy. They either rely on expensive laser equipment (laser-based online reconstruction), fail to guarantee reconstruction accuracy (image-based online reconstruction), or suffer from low reconstruction efficiency (overall offline reconstruction). When local scene updates or boundary expansion are needed, some reconstruction solutions lack suitable 3D scene update mechanisms. Update solutions that reconstruct from scratch (by re-acquiring data) suffer from low efficiency, poor scalability, and reliance on homogeneous data.

[0078] Based on the regional offline reconstruction scheme, the embodiments of this application inherit its advantages of requiring only images as input and achieving high reconstruction accuracy. Furthermore, by redesigning the optimization framework and function, and while satisfying the relative constraints of neighboring cameras in the global space, the camera trajectory alignment problem in the aggregation subclassing process is fundamentally solved, significantly accelerating the convergence speed and achieving more accurate large-scene reconstruction. This achieves a balance in terms of cost, speed, and accuracy. Based on regional reconstruction, the embodiments of this application design a hierarchical and decoupled optimization scheme. On the one hand, this scheme is naturally suitable for local replacement or outward expansion of existing scenes; on the other hand, it supports the use of heterogeneous data while ensuring the accuracy and consistency of the updated 3D scene.

[0079] Based on the regional offline reconstruction scheme, the embodiments of this application design a new 3D scene reconstruction system, which has a faster reconstruction speed and solves the problems of camera trajectory breakage and 3D point ghosting that are easy to occur when reconstructing aggregated sub-regions after regional reconstruction. In addition, it has a natural mechanism for updating 3D scenes (replacement and expansion) and also supports the use of heterogeneous data.

[0080] like Figure 1 As shown, the core of each embodiment of this application is a novel hierarchical decoupling optimization framework. Specifically, given all the data of a real-world scene, after matching, it is divided into multiple data subsets, corresponding to multiple local scenes. A local scene is called a subgraph. The common data points (such as images) between spatially adjacent subgraphs are called inter nodes, and the nodes inside a subgraph are called intra nodes. The hierarchical decoupling optimization framework has the following two key points:

[0081] 1) Treat each subgraph as a whole and rely on multiple inter nodes between spatially adjacent subgraphs to calculate the transformation relationship between subgraphs, such as scale transformation and similarity transformation. This calculation is achieved by considering the entire scene globally, constructing constraint relationships and establishing optimization problems based on the transformations (scale transformation and similarity transformation) between all spatially adjacent subgraphs. For example, a system of linear equations is constructed based on the relationship between absolute and relative poses, and then the absolute scale, absolute rotation, and coordinates of the origin of the (local) coordinate system of each subgraph in the global coordinate system are solved by minimizing the L1 loss.

[0082] 2) Fix the scale transformation and similarity transformation from the subgraph coordinate system to the global coordinate system (i.e., the absolute pose of the origin of the subgraph coordinate system in the global coordinate system), fix the pose of the intra nodes of the subgraph locally, and optimize the absolute pose of the inter nodes of the subgraph in the global coordinate system. The constraint relationship and optimization problem can be constructed by the relative pose between inter and intra in each subgraph.

[0083] As can be seen from the two key points above, the framework constructs optimization problems for solving at both the subgraph abstraction level and the subgraph internal level (data point fine-grained level), and the optimization at the two levels is decoupled from each other. Under this design, the optimization problem at each level is well-defined, and the scale of the optimization terms and parameters is controllable. Therefore, compared with other methods in this field, it has better convergence and faster convergence speed, thereby improving both reconstruction speed and reconstruction accuracy. The aforementioned subgraph can be a local scene divided from the overall scene, a local scene that needs to be replaced, or a newly expanded local scene. Therefore, this framework is suitable for the initial reconstruction of a 3D scene and naturally supports the updating and expansion of existing 3D scenes. At the same time, due to its hierarchical and decoupled characteristics, in the case of heterogeneous data, the subgraph can be reconstructed independently. Then, the relative poses between local regions are calculated based on the absolute poses of the common inter nodes between all spatially adjacent local regions. This serves as a constraint for the global optimization of the local coordinate system of the subgraph. Global optimization yields the absolute poses of the local regions in the global context, thereby ensuring the accuracy of the integration of local scenes into the overall scene. In the case of homogeneous data, further optimization can be performed at the data point level, further fine-tuning the global poses of data points at the intersections of local scenes.

[0084] The embodiments of this application can be deployed in the cloud as a cloud platform, or they can be deployed offline or as an integrated hardware and software machine. They can also be a set of cloud service application program interfaces (APIs) that work together.

[0085] The embodiments of this application can be applied to outdoor (parks, urban and rural streets, buildings, etc.) and indoor scene reconstruction. A panoramic camera can be used to collect on-site images (the red trajectory in the above figure is the trajectory of the camera during the acquisition process), or other photography equipment or laser equipment can be used to collect data for three-dimensional scene reconstruction.

[0086] The embodiments of this application have various possible product forms, including but not limited to cloud platforms, desktop software, integrated hardware and software systems, or a set of mutually cooperating cloud service APIs. Under the above-mentioned product forms, the system mainly provides users with functions such as data upload / import, user-defined update areas, 3D scene updates (replacement and expansion), large-scale new area reconstruction from scratch, hierarchical optimization options (abstract subgraph granularity, fine camera granularity optimization), and visualization of hierarchical optimization results (camera trajectory and 3D points).

[0087] The embodiments of this application are applicable to, but not limited to, mapping using panoramic camera acquisition methods. The embodiments of this application are also applicable to mapping image information obtained from different acquisition devices such as laser, LiDAR, depth cameras, monocular cameras, and binocular cameras, with different feature matching algorithms selected according to different acquisition methods. Furthermore, the optimization method of adding sensors based on the embodiments of this application is also conceivable; that is, by fusing data from multiple sensors, and based on the reconstruction scheme of the embodiments of this application, pose information is calculated and optimized to obtain a more accurate global pose based on multiple sensors.

[0088] The following is for reference Figure 2 and Figure 3 A three-dimensional reconstruction method according to an embodiment of this application is described. This three-dimensional reconstruction method can be performed by a computing device.

[0089] like Figure 2 As shown, according to this embodiment, the three-dimensional reconstruction method includes steps S210-S250. Each step will be described in detail below.

[0090] S210. Acquire scene data collected by the acquisition device for the original scene.

[0091] The original scene can be an initial scene that has not yet been updated, expanded, or changed. The purpose of 3D reconstruction is to first perform 3D modeling on the original scene.

[0092] The data acquisition device can be any device capable of collecting information about the shape, appearance, and location of a scene. Common data acquisition devices for 3D reconstruction include inertial sensors, LiDAR, ultrasonic radar, millimeter-wave radar, visible light cameras, and infrared cameras. Inertial sensors acquire data about the orientation and location of the scene or a portion thereof; LiDAR acquires point clouds; visible light cameras acquire images; and infrared cameras acquire infrared images. Visible light cameras can be further categorized into panoramic cameras, depth cameras, monocular cameras, and binocular cameras.

[0093] S220. Divide the scene data into multiple subclasses. Each subclass includes internal nodes specific to the subclass and common nodes shared with adjacent subclasses. The subclasses are collections of images, voxels, point clouds, and / or meshes.

[0094] A subclass (or subgraph) can refer to a set of data in scene data that has certain similarities or proximity. When the data is a 2D image, a subclass can be a set of multiple 2D images. When the data is a laser point cloud, a subclass can be a set of multiple 3D points (or multiple point cloud regions). When the data is a 3D image, a subclass can be a set of multiple voxels. When the data is data acquired by millimeter-wave radar, ultrasonic radar, and / or inertial sensors or other acquisition devices, a subclass can be a set of multiple grids.

[0095] An internal node (intra node) is a data node that exists exclusively within a specific subclass. A common node (inter node) is a data node shared between two adjacent subclasses. When the data is an image (visible light image, infrared image, etc.), the node can be the image itself. When the data is a point cloud, the node can contain the 3D coordinates of multiple points.

[0096] Scene data can be divided into multiple subclasses using graph cut algorithms.

[0097] Graph cut algorithms can be seen as associating image segmentation with the minimum cut problem of a graph. First, the image is mapped to a weighted undirected graph, where each node corresponds to a pixel in the image, and each edge connects a pair of adjacent pixels. The edge weight represents the non-negative similarity between adjacent pixels in terms of grayscale, color, or texture. Segmenting the image is equivalent to cutting the graph, with each segmented region corresponding to a subgraph. For example, the basic idea of ​​graph cut is to use the maximum flow algorithm to find the minimum cut, dividing the elements into two disjoint subsets. For example, graph cut is an operation for segmenting a weighted directed graph in a data structure. For example, graph cut can refer to segmenting an image into foreground and background using the minimum cut maximum flow algorithm.

[0098] When the scene data is an image, the graph cut process can be as follows: first, extract feature points from all images in the entire scene, perform image matching based on feature point matching, construct an edge graph, and then use the maximum flow algorithm to find the minimum cut, dividing the entire EG into multiple subgraphs.

[0099] S230. Based on the common nodes between every two adjacent subclasses in multiple subclasses, determine the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses.

[0100] Geometric transformations involve the interchange of two coordinate systems through geometric changes (scaling, rotation, translation, etc.). Relative geometric transformations can represent the relative scale transformations, relative rotation transformations, and relative translation transformations between two subclasses relative to each other. Absolute geometric transformations can represent the absolute scale transformations, absolute rotation transformations, and absolute translation transformations of a subclass relative to the global coordinate system or the entire 3D model.

[0101] A local coordinate system can refer to the coordinate system represented by all data (images, point clouds, etc.) within a subclass. When the data of the subclass is an image, the local coordinate system of the subclass represents a coordinate system that can determine the relative positions of pixels in all (2D) images contained in this subclass. For example, a local coordinate system can be a coordinate system with the optical center of the camera that took the image as the origin and the optical axis as the x-axis. When the subclass is a point cloud, the local coordinate system of the subclass can be a coordinate system that can determine the relative positions of all 3D points in this subclass.

[0102] Based on the common node between each pair of adjacent subclasses in multiple subclasses, the relative geometric transformation relationship between the local coordinate systems of the two adjacent subclasses can be determined. Specifically, since the pose or geometric transformation of (a) common node relative to the local coordinate system of each of the two adjacent subclasses is known, the relative geometric transformation relationship between the two subclasses can be obtained by using this common node as a bridge.

[0103] There can be multiple common nodes between any two adjacent subclasses. In this case, based on the multiple common nodes between any two adjacent subclasses, multiple relative geometric transformation relationships that constitute a set of relative geometric transformation relationships between the local coordinate systems of the two adjacent subclasses can be determined.

[0104] Typically, two adjacent subclasses share multiple common nodes, resulting in multiple relative geometric transformation relationships calculated from these common nodes. Therefore, relying on any single relative geometric transformation relationship to determine the true relative geometric transformation relationship between the two subclasses will be inaccurate. For ease of explanation, the multiple relative geometric transformation relationships calculated from multiple common nodes between two adjacent subclasses can be considered as a single set of relative geometric transformation relationships.

[0105] S240. Based on the relative geometric transformation relationships of multiple subclasses, determine the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system.

[0106] The relative geometric transformation relationship among multiple subclasses can refer to all the relative geometric transformation relationships existing in the subclasses formed by all the collected scene data, or it can refer to the relative geometric transformation relationship between all the subclasses formed by the scene data that needs to be modeled.

[0107] The global coordinate system can be determined as needed. For example, a local coordinate system of any subclass can be chosen as the global coordinate system, and then the transformation relationships of all other local coordinate systems relative to this global coordinate system can be solved.

[0108] Based on the relative geometric transformation relationships of multiple subclasses, the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system can be determined. Specifically, this can be achieved by substituting all relative geometric transformation relationships into a system of equations, then optimizing this system of equations using a loss function to determine the unique relative geometric transformation relationship between all adjacent subclasses. Finally, the absolute geometric transformation relationship of all local coordinate systems can be obtained based on these unique relative geometric transformation relationships.

[0109] When two adjacent subclasses share multiple common nodes, the absolute geometric transformation relationship of the subclass's local coordinate system relative to the global coordinate system can be determined based on multiple sets of relative geometric transformation relationships among the subclasses. Multiple sets of relative geometric transformation relationships among the subclasses can refer to the relative geometric transformation relationships between all adjacent subclasses across all subclasses, with each pair of adjacent subclasses having one set of relative geometric transformation relationships. Thus, all subclasses (or all subclasses of interest) have multiple sets of relative geometric transformation relationships.

[0110] Based on multiple sets of relative geometric transformation relationships among various subclasses, the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system can be determined using the L1 loss function. Specifically, the L1 loss function can be used to calculate this relationship by substituting all sets of relative geometric transformation relationships from all subclasses into a system of equations. The L1 loss function is then used to find the solution to this system of equations that minimizes the loss. The solution to this system of equations then uniquely determines the relative geometric transformation relationship between any two adjacent subclasses.

[0111] In S240, each subclass is treated as a node (i.e., the origin of the subclass coordinate system). Taking into account the entire scene, the constraints (scale transformation and similarity transformation) between all subclasses calculated in the third step are added to a system of equations. The absolute scale, absolute rotation, and coordinates of the origin of each subclass coordinate system in the global coordinate system are solved by L1 loss optimization, that is, the scale transformation and similarity transformation of the subclass pose to the global coordinate system are obtained.

[0112] The calculation process of S240 is described in detail below.

[0113] Based on the relative scale calculated between inter nodes of subclasses, the following equation is obtained:

[0114] s i / s j =S ij

[0115] Taking the logarithm of both sides, we get:

[0116] log(s i )-log(sj ) = log(S ij )

[0117] The equations obtained by summing up all relative scales between adjacent subclasses in space are then obtained.

[0118] A s x s =b s

[0119] Among them, A s It is a sparse matrix, where each row has one element that is 1, one element that is -1, and all other elements that are 0. s It is the vector obtained by taking the logarithm of the absolute scale of all subclasses, b s It is a vector obtained by taking the logarithm of the relative scales.

[0120] By setting the absolute scale of the first subclass to 1, i.e., log(s1) = 0, the above system of equations can be solved by solving the following convex L1 optimization:

[0121]

[0122] Similarly, the relative rotations between spatially adjacent subclasses are computed, and the absolute rotation of each subclass in the global coordinate system is estimated using the Rotation Averaging method.

[0123] Similarly, by calculating the absolute scale of each subclass, the relative translations between spatially neighboring subclasses in the global coordinate system are calculated through the inter nodes, resulting in the following equation:

[0124] R j (c i -c j )=t ij

[0125] Among them, R j It is the absolute rotation of subclass j in the global coordinate system, c i c j The coordinates of the origin of the subclass i coordinate system and the origin of the subclass i coordinate system in the global coordinate system are t. ij It is the relative translation between the origin of the global coordinate system subclass i and the origin of the global coordinate system subclass j.

[0126] Summarizing the equations obtained from all relative translations between adjacent subclasses in space, and stacking them up, we get:

[0127] A c x c =b c

[0128] Where A cIt is a sparse matrix, where every three consecutive rows contain R. j and -R j The rest are all 0, x c It is a vector obtained by arranging the coordinates of the origins of all subclass coordinate systems in the global coordinate system, b c It is a vector obtained by arranging the relative translations between subclasses.

[0129] Similarly, fixing the coordinates of the origin of the first subclass coordinate system to 0 in the global coordinate system, i.e., c1 = 0, the above system of equations is solved by solving the following convex L1 optimization:

[0130]

[0131] Based on the above calculation process, S240 has the following advantages:

[0132] 1) Accuracy: Treat each subclass as a node, calculate multiple relative scales, relative rotations, and relative translations between spatially adjacent subclasses through inter nodes. Since there are multiple inter nodes between any two spatially adjacent subclasses, multiple relative scale pose constraints will be obtained. No single constraint is absolutely accurate, but when they are combined, a more robust solution that satisfies all constraints will be obtained. At the same time, global optimization summarizes the relative scales and pose constraints between all spatially adjacent subclasses in the entire scene, and solves the global optimal solution that satisfies all constraints.

[0133] 2) High efficiency: Because the subclass is treated as a node (the origin of the subclass coordinate system) for global optimization, it is more efficient and the optimization is faster.

[0134] Figure 3 In the visual representation S240, each subclass is treated as a node. Global L1 loss is used to optimize the global absolute scale and similarity transformation of each subclass, aggregating the results from the subclasses to the global domain. For example... Figure 3 As shown, the improvements in this embodiment are: 1) the camera trajectories of the connection positions between subclasses are well aligned; 2) the panoramic camera pose used during acquisition is basically reproduced in the trajectory (eight cameras facing the four directions around a point); 3) the spatial 3D points estimated by each subclass are aligned, eliminating the ghosting problem.

[0135] For example, S240 may further include optimizing the absolute geometric transformation relationship of common nodes. That is, fixing the scale transformation, similarity transformation, and absolute pose of intra connected to inter in the subclass in the local area, and optimizing the absolute pose of inter in the global coordinate system by relying on the relative pose constraints between local inter and intra.

[0136] Specifically, optimize the following two loss functions:

[0137]

[0138] Where Rij is the relative rotation of the inter-intra subclass in the global coordinate system. It is the absolute rotation of camera j in the global coordinate system. It is the absolute rotation of camera i in the local coordinate system A. It is a rotational transformation from the local coordinate system A to the global coordinate system B.

[0139]

[0140] Among them, t ij It is the relative translation between inter-intra cameras in the local subclass after the relative translation transformation to the global domain (actually equal to the local relative translation multiplied by the absolute scale of the subclass in the global coordinate system). It is the absolute rotation of camera j in the global coordinate system. These are the coordinates of camera j in the global coordinate system. These are the coordinates of camera i in the local coordinate system A. These are the coordinates of the origin of the local subclass A coordinate system in the global B coordinate system. It is a rotational transformation from the local coordinate system A to the global coordinate system B.

[0141] Further optimization of the absolute geometric transformation relationship of common nodes has the following advantages:

[0142] 1) High efficiency: Because the scale transformation, similarity transformation, and absolute pose of intra nodes in the local coordinate system are fixed from the subclass to the global domain, the absolute pose of intra nodes connected to inter in each subclass in the space is kept fixed in the global coordinate system. This is equivalent to having multiple data points in each local region with fixed poses in the global coordinate system. Only the poses of inter nodes in the subclass in the global coordinate system are optimized, which greatly speeds up the convergence speed and optimizes very quickly.

[0143] 2) Accuracy: Because the scale and similarity transformations from subclasses to the global domain remain unchanged, for the same inter, the relative constraints of multiple inter-intra nodes in adjacent subclasses are added to transform them to the global coordinate system, thus optimizing the pose of the inter nodes in the global coordinate system. Therefore, the optimization further improves the pose of the inter nodes while maintaining the good global optimization results of the abstract layer in the hierarchical decoupling.

[0144] S250. Based on the absolute geometric transformation relationships of multiple subclasses, construct a 3D model of the original scene.

[0145] Since the absolute geometric transformation relationship of the local coordinate system of each subclass relative to the global coordinate system is known, the absolute position of the pixel in the data or image of each subclass in the global coordinate system can be determined by this transformation relationship. By combining the absolute positions of all data, a point cloud model in three-dimensional space (such as sparse point cloud) can be formed. Further optimization of the point cloud model can yield a three-dimensional model.

[0146] Specifically, the pose and 3D points of the acquisition device of the subclass in the global coordinate system can be determined first based on the absolute geometric transformation relationship of the subclass. Then, a 3D model of the original scene can be constructed based on the pose and 3D points of the acquisition devices of multiple subclasses.

[0147] The pose of a data acquisition device refers to its position and orientation (including rotation, translation, etc.) relative to a coordinate system when acquiring data. For example, when the data is an image, the pose of the acquisition device can represent its position and rotation angle when capturing an image. A 3D point can refer to a point with positional information in a global coordinate system reconstructed from the data or image acquired by the acquisition device.

[0148] The poses and 3D points of multiple subclasses of acquisition devices can refer to the poses and 3D points of all acquisition devices obtained by calculation for all subclasses (or all subclasses that need to be focused on). A point cloud model can be reconstructed from these poses and 3D points, and a complete 3D model can be constructed based on this point cloud model.

[0149] The following is for reference Figure 4 This application describes a three-dimensional reconstruction method according to another embodiment. This embodiment proposes a method for... Figure 2 The method described in this embodiment updates the original scene 3D model constructed by the 3D reconstruction method.

[0150] According to this embodiment, any local area of ​​any size in the reconstructed scene (where the scene changes due to road construction, seasonal changes, site renovation, interior decoration, etc.) can be updated.

[0151] First, we will provide a general overview of the operation flow of the update method according to this embodiment.

[0152] First, the regions requiring updates are removed from the reconstructed scene. All adjacent intra nodes within the user-drawn boundary are treated as a subclass C0, and the global scale and pose of the C0 coordinate system are fixed. Then, for the regions requiring updates, data (images, laser point clouds) is reacquired on-site. An EG (Graphical Ensemble) is constructed using feature matching on the new data, and graph cuts are used to obtain sub-clusters. These sub-clusters are then expanded to generate shared inter nodes for adjacent subclasses, and reconstruction is performed on all subclasses in parallel. Next, each subclass (including C0) is treated as a node. The scale and similarity transformations between adjacent subclasses are calculated using the inter nodes. Based on the constraints between all adjacent subclasses in space, the absolute scale and pose of each subclass in the global coordinate system are globally optimized, thus aggregating the subclasses into the reconstructed scene (or aggregating them together if reconstructing from scratch). If fine-tuning of the global pose of shared data points between subclasses is required, the pose of shared data points can be optimized using the second layer of the hierarchical decoupling framework.

[0153] like Figure 4 As shown, the three-dimensional reconstruction method according to this embodiment includes steps S401-S412. Each step will be described in detail below.

[0154] S401. Acquire scene data collected by the acquisition device for the original scene.

[0155] S402. Divide the scene data into multiple subclasses. Each subclass includes internal nodes that are unique to the subclass and common nodes that are shared with adjacent subclasses.

[0156] S403. Based on the common nodes between any two adjacent subclasses in multiple subclasses, determine the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses.

[0157] S404. Based on the relative geometric transformation relationships of multiple subclasses, determine the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system.

[0158] S405. Based on the absolute geometric transformation relationships of multiple subclasses, construct a 3D model of the original scene.

[0159] For details regarding S401 to S405, please refer to the above section on... Figure 2 The detailed description of the embodiments is omitted here.

[0160] S406. Obtain the model area to be updated drawn by the user on the 3D model of the original scene.

[0161] The model region can refer to a portion of the 3D model reconstructed from the original scene, which needs to be updated.

[0162] When the model needs to be updated, the user can determine which part of the 3D model needs to be updated or replaced. The user can then operate on the 3D model to outline the part that needs to be updated.

[0163] S407. Delete the model area to be updated.

[0164] Once a portion of the model that needs updating has been drawn, this area can be deleted directly, and the remaining part becomes the remaining model area. There is a boundary (boundary line) between the deleted area and the remaining area.

[0165] S408. Merge the scene data of the remaining model area into the pre-update subclass.

[0166] The remaining model region is constructed based on the data previously collected for the original scene. The relative and absolute geometric transformation relationships of this part of the model and the subclasses used to construct it have been optimized and calculated. Therefore, all the scene data used to construct the remaining model region can be directly merged into a subclass, namely the pre-update subclass, for subsequent calculations.

[0167] S409. Divide the scene data of the updated scene into multiple update subclasses.

[0168] Updating a scene can refer to the part of the scene that has changed relative to the original scene, and this part of the scene needs to replace the deleted scene model.

[0169] The scene data collected for the updated scene has not yet been optimized, so it needs to be re-segmented into multiple subclasses in order to construct an accurate 3D model of the updated scene based on this scene data.

[0170] S410. Select the common node between the pre-update subclass and the updated subclass on the boundary of the remaining model region.

[0171] The boundary of the remaining model region can refer to the boundary between the deleted model region and the remaining model region. Since the model is built based on the data or nodes of the subclasses, there must be subclass nodes on this boundary. By selecting these nodes, some of them can be used as common nodes to connect the unupdated subclass and one of the updated subclasses.

[0172] S411. Merge the pre-update subclass and multiple updated subclasses into multiple post-update subclasses.

[0173] After a common node is selected, the unupdated subclass and one of the updated subclasses are linked together through the common node, becoming adjacent subclasses. The unupdated subclass and all updated subclasses are then merged together to form a new subclass cluster, i.e., multiple updated subclasses.

[0174] S412. Based on multiple updated subclasses, construct a 3D model of the updated scene.

[0175] Based on multiple updated subclasses, construct a 3D model of the updated scene. The specific method can be found in [reference needed]. Figure 1 or Figure 2 Description of the embodiment. For example, firstly, based on the common nodes between every two adjacent updated subclasses in multiple updated subclasses, the relative geometric transformation relationship between the local coordinate systems of two adjacent updated subclasses is determined. Then, based on the relative geometric transformation relationship of multiple updated subclasses, the absolute geometric transformation relationship of the local coordinate system of the updated subclasses relative to the global coordinate system is determined. Finally, based on the absolute geometric transformation relationship of multiple updated subclasses, a 3D model of the updated scene is constructed.

[0176] The following is for reference Figure 5 This application describes a three-dimensional reconstruction method according to another embodiment. This embodiment proposes a method for... Figure 2 The method described in this embodiment expands the original scene 3D model constructed by the 3D reconstruction method.

[0177] According to this embodiment, the reconstructed scene can be expanded to add new scenes.

[0178] First, we will provide an overview of the operation flow of the extended method according to this embodiment.

[0179] Similar to the update method, we can update a portion of the reconstructed scene, treating intra cameras adjacent to the boundary in the reconstructed scene as subclass C0. When acquiring new data for the expanded region, we follow the same process as in the update scheme: graph cut to obtain subclasses, expansion to generate inter nodes -> calculate scale and similarity transformations between adjacent subclasses -> global optimization to solve for the absolute scale, absolute rotation, and absolute translation of the origin of each subclass's coordinate system in the global coordinate system -> aggregate subclasses -> optimize the absolute pose of the subclass inter nodes in the global coordinate system.

[0180] like Figure 5 As shown, the three-dimensional reconstruction method of this embodiment includes steps S501-S511. Each step will be described in detail below.

[0181] S501. Acquire scene data collected by the acquisition device for the original scene.

[0182] S502. Divide the scene data into multiple subclasses. Each subclass includes internal nodes that are unique to the subclass and common nodes that are shared with adjacent subclasses.

[0183] S503. Based on the common nodes between any two adjacent subclasses in multiple subclasses, determine the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses.

[0184] S504. Based on the relative geometric transformation relationships of multiple subclasses, determine the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system.

[0185] S505. Based on the absolute geometric transformation relationships of multiple subclasses, construct a 3D model of the original scene.

[0186] For details regarding S501 to S505, please refer to the above section on... Figure 2 The detailed description of the embodiments is omitted here.

[0187] S506. Obtain the boundary between the 3D model of the extended scene and the 3D model of the original scene.

[0188] An extended scene can refer to a scene that needs to be added on top of the original scene, or it can be a scene of a geographical area adjacent to the original scene.

[0189] An extended scene is generally connected to the original scene, so the boundary between them can be defined. This boundary, in a 3D model, is the boundary between the 3D model of the extended scene and the 3D model of the original scene. Before the 3D model of the extended scene is built, this boundary can be determined by sketching or selecting a portion of the 3D model of the original scene.

[0190] S507. Merge the scene data of the original scene into an extended subclass.

[0191] The scene data of the original scene has been optimized and calculated to form a stable and reliable 3D model. Therefore, there is no need to further segment the original scene data. All scene data collected for the original scene can be directly merged into a subclass, namely the pre-extension subclass.

[0192] S508. Divide the scene data of the extended scene into multiple extended subclasses.

[0193] The scene data collected for the extended scenarios has not yet been calculated and optimized. It needs to be divided into multiple extended subclasses according to the previous ideas and algorithms (e.g., through graph cut algorithms).

[0194] S509. Select the common node between the pre-extended subclass and the extended subclass on the boundary.

[0195] Since the location of the boundary between the extended scene and the original scene on the original 3D model is known, we can select data nodes that can be used as common nodes between the original subclass and one of the extended subclasses for this part of the 3D model.

[0196] S510. Merge the pre-extension subclass and multiple extended subclasses into multiple post-extension subclasses.

[0197] Once a pre-extension subclass and one of the extended subclasses are linked through a selected common node, the pre-extension subclass can be combined with all the extended subclasses to form a new subclass cluster, i.e., multiple extended subclasses.

[0198] S511. Based on multiple extended subclasses, construct a 3D model of the extended scene.

[0199] Based on multiple extended subclasses, construct the 3D model of the extended scene. The specific method can be found in [reference needed]. Figure 1 or Figure 2 Description of the embodiment. For example, firstly, based on the common nodes between every two adjacent extended subclasses in multiple extended subclasses, the relative geometric transformation relationship between the local coordinate systems of two adjacent extended subclasses is determined. Then, based on the relative geometric transformation relationship of multiple extended subclasses, the absolute geometric transformation relationship of the local coordinate system of the extended subclasses relative to the global coordinate system is determined. Finally, based on the absolute geometric transformation relationship of multiple extended subclasses, a 3D model of the extended scene is constructed.

[0200] Based on the foregoing Figure 2 The embodiments of the method described in this application also provide a three-dimensional reconstruction device, the structural schematic diagram of which is shown below. Figure 6 As shown. This three-dimensional reconstruction device is used to perform the aforementioned... Figure 2 Each step in the process.

[0201] According to this embodiment, the three-dimensional reconstruction device 600 includes: a first acquisition module 610, a first segmentation module 620, a first determination module 630, a second determination module 640, and a first construction module 650.

[0202] The first acquisition module 610 is used to acquire scene data collected by the acquisition device for the original scene.

[0203] The first segmentation module 620 is used to divide the scene data into multiple subclasses. Each subclass includes internal nodes specific to the subclass and common nodes shared with adjacent subclasses. The subclasses are collections of images, voxels, point clouds, and / or meshes.

[0204] The first determining module 630 is used to determine the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses based on the common nodes between every two adjacent subclasses in multiple subclasses.

[0205] The second determining module 640 is used to determine the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system based on the relative geometric transformation relationship of multiple subclasses.

[0206] The first construction module 650 is used to construct a 3D model of the original scene based on the absolute geometric transformation relationships of multiple subclasses.

[0207] Based on the foregoing Figure 4 The embodiments of the method described in this application also provide a three-dimensional reconstruction device, the structural schematic diagram of which is shown below. Figure 7 As shown. This three-dimensional reconstruction device is used to perform the aforementioned... Figure 4 Each step in the process.

[0208] According to this embodiment, the three-dimensional reconstruction device 700 includes: a first acquisition module 701, a first segmentation module 702, a first determination module 703, a second determination module 704, a first construction module 705, a second acquisition module 706, a deletion module 707, a first merging module 708, a second segmentation module 709, a first selection module 710, a second merging module 711, and a second construction module 712.

[0209] The first acquisition module 701 is used to acquire scene data collected by the acquisition device for the original scene.

[0210] The first segmentation module 702 is used to divide the scene data into multiple subclasses. Each subclass includes internal nodes specific to the subclass and common nodes shared with adjacent subclasses. The subclasses are collections of images, voxels, point clouds, and / or meshes.

[0211] The first determining module 703 is used to determine the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses based on the common nodes between every two adjacent subclasses in multiple subclasses.

[0212] The second determining module 704 is used to determine the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system based on the relative geometric transformation relationship of multiple subclasses.

[0213] The first construction module 705 is used to construct a 3D model of the original scene based on the absolute geometric transformation relationships of multiple subclasses.

[0214] The second acquisition module 706 is used to acquire the model area to be updated drawn by the user on the 3D model of the original scene.

[0215] The deletion module 707 is used to delete the model region to be updated.

[0216] The first merging module 708 is used to merge the scene data of the remaining model regions into a pre-update subclass.

[0217] The second segmentation module 709 is used to divide the scene data of the updated scene into multiple update subclasses.

[0218] The first selection module 710 is used to select common nodes between the pre-update subclass and the updated subclass on the boundary of the remaining model region.

[0219] The second merging module 711 is used to merge the pre-update subclass and multiple updated subclasses into multiple post-update subclasses.

[0220] The second construction module 712 is used to construct a 3D model of the updated scene based on multiple updated subclasses.

[0221] Based on the foregoing Figure 5 The embodiments of the method described in this application also provide a three-dimensional reconstruction device, the structural schematic diagram of which is shown below. Figure 8 As shown. This three-dimensional reconstruction device is used to perform the aforementioned... Figure 5 Each step in the process.

[0222] According to this embodiment, the three-dimensional reconstruction device 800 includes: a first acquisition module 801, a first segmentation module 802, a first determination module 803, a second determination module 804, a first construction module 805, a third acquisition module 806, a third merging module 807, a third segmentation module 808, a second selection module 809, a fourth merging module 810, and a third construction module 811.

[0223] The first acquisition module 801 is used to acquire scene data collected by the acquisition device for the original scene.

[0224] The first segmentation module 802 is used to divide the scene data into multiple subclasses. Each subclass includes internal nodes that are exclusive to the subclass and common nodes that are shared with adjacent subclasses. The subclasses are a collection of images, voxels, point clouds and / or meshes.

[0225] The first determining module 803 is used to determine the relative geometric transformation relationship between the local coordinate systems of two adjacent subclasses based on the common nodes between every two adjacent subclasses in multiple subclasses.

[0226] The second determining module 804 is used to determine the absolute geometric transformation relationship of the local coordinate system of a subclass relative to the global coordinate system based on the relative geometric transformation relationship of multiple subclasses.

[0227] The first construction module 805 is used to construct a 3D model of the original scene based on the absolute geometric transformation relationships of multiple subclasses.

[0228] The third acquisition module 806 is used to acquire the boundary between the 3D model of the extended scene and the 3D model of the original scene.

[0229] The third merging module 807 is used to merge the scene data of the original scene into an extended subclass.

[0230] The third segmentation module 808 is used to divide the scene data of the extended scene into multiple extended subclasses.

[0231] The second selection module 809 is used to select a common node between the pre-extended subclass and the extended subclass on the boundary.

[0232] The fourth merging module 810 is used to merge the pre-extension subclass and the multiple extended subclasses into multiple post-extension subclasses.

[0233] The third construction module 811 is used to construct a 3D model of the extended scene based on the multiple extended subclasses.

[0234] It should be noted that, Figure 6 , Figure 7 and Figure 8 The three-dimensional reconstruction devices 600, 700, and 800 provided in the illustrated embodiments, when performing the three-dimensional reconstruction method, are only illustrated by the division of the above-described functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the three-dimensional reconstruction devices 600, 700, and 800 provided in the above embodiments and... Figure 2 , Figure 4 and Figure 5 The three-dimensional reconstruction method embodiments shown are all based on the same concept, and their specific implementation processes are detailed in the method embodiments, which will not be repeated here.

[0235] Figure 9 This is a schematic diagram of the hardware structure of a computing device 900 provided in an embodiment of this application.

[0236] See Figure 9 The computing device 900 includes a processor 910, a memory 920, a communication interface 930, and a bus 940, which are interconnected via the bus 940. The processor 910, memory 920, and communication interface 930 may also be connected using other methods besides the bus 940.

[0237] The memory 920 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, hard disk, etc.

[0238] The processor 910 may be a general-purpose processor, which can be a processor that performs specific steps and / or operations by reading and executing contents stored in memory (e.g., memory 920). For example, the general-purpose processor may be a central processing unit (CPU). The processor 910 may include at least one circuit to perform... Figure 2 , Figure 4 and Figure 5 The illustrated embodiments provide all or part of the steps of the three-dimensional reconstruction method.

[0239] The communication interface 930 includes input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the computing device 900, as well as interfaces for interconnecting the computing device 900 with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc. The communication interface 930 can connect to external input and output devices. For example, input devices can be microphones or microphone arrays for capturing voice input signals; they can be communication network connectors for receiving collected input signals from the cloud or other devices; and they can also include, for example, keyboards, mice, etc. Output devices can output various information externally, including determined distance information, direction information, etc. Output devices can include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, etc.

[0240] The bus 940 can be any type of communication bus used to interconnect the processor 910, memory 920 and communication interface 930, such as a system bus.

[0241] The aforementioned devices can be disposed on separate chips, or at least partially or entirely on the same chip. Whether to dispose of the devices independently on different chips or integrate them on one or more chips often depends on the needs of the product design. This application does not limit the specific implementation of the aforementioned devices.

[0242] Figure 9 The computing device 900 shown is merely an example. In the implementation process, the computing device 900 may also include other components, which will not be listed one by one in this article.

[0243] Embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the three-dimensional reconstruction methods according to various embodiments of this application described above.

[0244] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0245] The concepts, principles, and ideas of this application have been described in detail above with reference to specific embodiments (including examples and instances). Those skilled in the art should understand that the embodiments of this application are not limited to those given above. After reading this application, those skilled in the art can make any possible improvements, substitutions, and equivalents to the steps, methods, apparatus, and components in the above embodiments, and such improvements, substitutions, and equivalents should be considered to fall within the scope of this application. The scope of protection of this application is limited to the claims.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Acquire scene data collected by the acquisition device for the original scene; The scene data is divided into multiple subclasses. Each subclass includes internal nodes specific to the subclass and multiple common nodes shared with adjacent subclasses. The subclasses are collections of images, voxels, point clouds, and / or meshes. Based on the multiple common nodes between every two adjacent subclasses, determine multiple relative geometric transformation relationships between the local coordinate systems of the two adjacent subclasses, which constitute a set of relative geometric transformation relationships; Based on the multiple sets of relative geometric transformation relationships of the multiple subclasses, the absolute geometric transformation relationship of the local coordinate system of the subclass relative to the global coordinate system is calculated using the L1 loss function; Based on the absolute geometric transformation relationships of the multiple subclasses, a 3D model of the original scene is constructed.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, The process of dividing the scene data into multiple subclasses includes: The scene data is divided into multiple subclasses using a graph cut algorithm.

3. The three-dimensional reconstruction method according to claim 1, characterized in that, The step of constructing a 3D model of the original scene based on the absolute geometric transformation relationships of the multiple subclasses includes: Based on the absolute geometric transformation relationship of the subclass, determine the acquisition device pose and 3D point of the subclass in the global coordinate system; Based on the poses and 3D points of the acquisition devices of the multiple subclasses, a 3D model of the original scene is constructed.

4. The three-dimensional reconstruction method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain the model area to be updated drawn by the user on the 3D model of the original scene; Delete the model region to be updated; Merge the scene data of the remaining model regions into a pre-update subclass; Divide the scene data of the updated scene into multiple update subclasses; Select the common node between the pre-update subclass and the updated subclass on the boundary of the remaining model region; The pre-update subclass and the multiple updated subclasses are merged into multiple post-update subclasses; Based on the multiple updated subclasses, construct a 3D model of the updated scene.

5. The three-dimensional reconstruction method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain the boundary between the 3D model of the extended scene and the 3D model of the original scene; The scene data of the original scene is merged into an extended subclass; Divide the scene data of the extended scene into multiple extended subclasses; Select the common node between the original subclass and the extended subclass on the boundary; The original subclass and the multiple extended subclasses are merged into multiple extended subclasses; Based on the multiple extended subclasses, construct a 3D model of the extended scene.

6. The three-dimensional reconstruction method according to any one of claims 1 to 3, characterized in that, The relative geometric transformation relationships include relative scale transformation relationships, relative rotation transformation relationships, and relative translation transformation relationships, and the absolute geometric transformation relationships include absolute scale transformation relationships, absolute rotation transformation relationships, and absolute translation transformation relationships.

7. The three-dimensional reconstruction method according to any one of claims 1 to 3, characterized in that, The data acquisition equipment includes one or more of the following: inertial sensors, lidar, ultrasonic radar, millimeter-wave radar, visible light cameras, and infrared cameras.

8. A three-dimensional reconstruction device, characterized in that, The device includes: The first acquisition module is used to acquire scene data collected by the acquisition device for the original scene; The first segmentation module is used to divide the scene data into multiple subclasses. Each subclass includes internal nodes specific to the subclass and multiple common nodes shared with adjacent subclasses. The subclasses are collections of images, voxels, point clouds, and / or meshes. The first determining module is used to determine, based on the multiple common nodes between each pair of adjacent subclasses, multiple relative geometric transformation relationships that constitute a set of relative geometric transformation relationships between the local coordinate systems of the two adjacent subclasses; The second determining module is used to calculate the absolute geometric transformation relationship of the local coordinate system of the subclass relative to the global coordinate system by using the L1 loss function based on the multiple sets of relative geometric transformation relationships of the multiple subclasses. The first construction module is used to construct a three-dimensional model of the original scene based on the absolute geometric transformation relationships of the multiple subclasses.

9. The three-dimensional reconstruction device according to claim 8, characterized in that, The first segmentation module is further configured to: The scene data is divided into multiple subclasses using a graph cut algorithm.

10. The three-dimensional reconstruction apparatus according to claim 8, characterized in that, The first building module is further configured to: Based on the absolute geometric transformation relationship of the subclass, determine the acquisition device pose and 3D point of the subclass in the global coordinate system; Based on the poses and 3D points of the acquisition devices of the multiple subclasses, a 3D model of the original scene is constructed.

11. The three-dimensional reconstruction apparatus according to any one of claims 8 to 10, characterized in that, The device further includes: The second acquisition module is used to acquire the model area to be updated drawn by the user on the 3D model of the original scene; The deletion module is used to delete the model region to be updated; The first merging module is used to merge the scene data of the remaining model regions into the pre-update subclass; The second segmentation module is used to divide the scene data of the updated scene into multiple update subclasses; The first selection module is used to select a common node between the pre-update subclass and the updated subclass on the boundary of the remaining model region; The second merging module is used to merge the pre-update subclass and the multiple updated subclasses into multiple post-update subclasses; The second construction module is used to construct a 3D model of the updated scene based on the multiple updated subclasses.

12. The three-dimensional reconstruction apparatus according to any one of claims 8 to 10, characterized in that, The device further includes: The third acquisition module is used to acquire the boundary between the 3D model of the extended scene and the 3D model of the original scene; The third merging module is used to merge the scene data of the original scene into an extended subclass; The third segmentation module is used to divide the scene data of the extended scene into multiple extended subclasses; The second selection module is used to select a common node between the pre-extended subclass and the extended subclass on the boundary; The fourth merging module is used to merge the pre-extension subclass and the multiple extended subclasses into multiple post-extension subclasses; The third construction module is used to construct a 3D model of the extended scene based on the multiple extended subclasses.

13. The three-dimensional reconstruction apparatus according to any one of claims 8 to 10, characterized in that, The relative geometric transformation relationships include relative scale transformation relationships, relative rotation transformation relationships, and relative translation transformation relationships, and the absolute geometric transformation relationships include absolute scale transformation relationships, absolute rotation transformation relationships, and absolute translation transformation relationships.

14. The three-dimensional reconstruction apparatus according to any one of claims 8 to 10, characterized in that, The data acquisition equipment includes one or more of the following: inertial sensors, lidar, ultrasonic radar, millimeter-wave radar, visible light cameras, and infrared cameras.

15. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the three-dimensional reconstruction method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for performing the three-dimensional reconstruction method according to any one of claims 1 to 7.

Images