Incremental generation method of global point cloud, three-dimensional reconstruction method and computing equipment

Abstract

The invention relates to a global point cloud increment generation method, a three-dimensional reconstruction method and computing equipment, and the method comprises the steps: generating a first local point cloud described by a first local reference system based on a first image sequence of a target object collected by a camera, a part of image frames in the first image sequence are reference image frames which establish mapping with a global point cloud fragment of the target object, and the global point cloud fragment is described by a global reference system; based on a camera pose corresponding relation of the reference image frame between the first local reference system and the global reference system, performing first-level registration on the first local point cloud to obtain an initial registration point cloud; based on the geometric attribute corresponding relation between the initial registration point cloud and the global point cloud fragment, second-level registration is carried out on the initial registration point cloud, and a first global point cloud fragment described by a global reference system is obtained; wherein the first global point cloud fragment is used for being fused with other global point cloud fragments of the target object to form a global point cloud of the target object.

Description

Technical Field

[0001] This disclosure relates to the field of computer processing technology, and more specifically, to a method for incremental generation of global point clouds, a three-dimensional reconstruction method, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of computer vision technology, 3D reconstruction technology has been widely used in fields such as virtual reality and autonomous driving. 3D reconstruction technology acquires surface geometric information of a target object from multiple angles and then reconstructs a 3D model of the target object using computer algorithms.

[0003] The foundation of high-precision 3D reconstruction lies in obtaining dense global point cloud data. The traditional method for generating global point cloud data is a two-tiered framework combining Structure from Motion (SFM) and Multi-view Stereo (MVS). Taking road reconstruction based on this framework in the field of autonomous driving as an example, SFM performs feature matching and global bundle adjustment (BA) on continuously acquired image sequences during vehicle movement to calculate the vehicle trajectory and generate a sparse point cloud; MVS, on the other hand, takes the SFM results as input, and outputs a dense point cloud through pixel-by-pixel depth estimation and multi-view fusion for extracting road features for 3D reconstruction.

[0004] While this framework can achieve high-precision 3D reconstruction, it suffers from serious technical problems such as excessive consumption of computational resources and high time costs. These problems are reflected in: 1) the computational complexity increases superlinearly with the resolution and number of images; 2) global bundle adjustment is extremely sensitive to memory bandwidth and is prone to slow convergence or local optima in large-scale scenes; 3) the dense matching stage requires searching for matching windows pixel by pixel, resulting in a large amount of redundant computation.

[0005] Therefore, due to limitations in algorithm architecture and computing resources, it is difficult to complete the computation within a reasonable time when processing large-scale data based on traditional methods, thus failing to meet application requirements. Summary of the Invention

[0006] In view of this, the present disclosure proposes a technical solution that can improve the generation efficiency of dense point clouds.

[0007] According to a first aspect of this disclosure, an incremental generation method for a global point cloud is provided, comprising:

[0008] Based on a first image sequence of a target object acquired by a camera, a first local point cloud is generated, described by a first local reference frame; wherein, a portion of the image frames in the first image sequence are reference image frames that establish a mapping with a global point cloud segment of the target object, and the global point cloud segment is described by a global reference frame;

[0009] Based on the camera pose correspondence between the reference image frame and the global reference frame in the first local reference frame, the first local point cloud is registered at the first level to obtain the initial registered point cloud.

[0010] Based on the geometric attribute correspondence between the initial registration point cloud and the global point cloud fragment, a second-level registration is performed on the initial registration point cloud to obtain a first global point cloud fragment of the target object described by the global reference system; wherein, the first global point cloud fragment is used to merge with other global point cloud fragments of the target object to form the global point cloud of the target object.

[0011] Optionally, according to the method of claim 1, the rotation transformation parameters of the first-level registration are determined by decoupling the rotation relationship from the camera pose correspondence of each of the reference image frames.

[0012] Optionally, the translation transformation parameters of the first-level registration are determined by correcting the translation relationship of each of the reference image frames in the camera pose relationship with the rotation transformation parameters.

[0013] Optionally, determining the rotation transformation parameters based on the rotation relationship includes:

[0014] The rotation transformation parameters are obtained based on the rotation relationship between the first camera pose in the first local reference frame and the second camera pose in the global reference frame for each reference image frame; and / or,

[0015] Correcting the translation relationship of each reference image frame in the camera pose relationship using the rotation transformation parameters to determine the translation transformation parameters for the first-level registration includes:

[0016] The rotation transformation parameters are applied to the translation vector of the first camera pose of the reference image frame to obtain the rotation-corrected translation vector of the reference image frame.

[0017] The translation transformation parameters are determined based on the deviation between the rotation-corrected translation vector of each reference image frame and the translation vector of the second camera pose of the corresponding reference image frame.

[0018] Optionally, generating a first local point cloud described in a first local reference frame based on the first image sequence of the target object acquired by the camera includes:

[0019] Based on the geometric consistency constraints between different image frames in the first image sequence, the first acquisition information of the first image sequence is obtained; wherein, the first acquisition information includes camera intrinsic parameters, as well as the scale-free camera pose and scale-free pixel depth of each image frame in the first image sequence.

[0020] Scale recovery is performed on the first acquired information to obtain second acquired information with scale.

[0021] Based on the second acquired information, the pixels in the first image sequence are mapped to three-dimensional space to obtain the first local point cloud.

[0022] Optionally, the first acquired information is obtained by learning geometric consistency regression between different image frames in the first image sequence using an end-to-end model.

[0023] Optionally, the scaling factor on which the scale recovery is based is determined based on the scaled sparse trajectory of the motion recovery structure.

[0024] Optionally, the geometric attribute correspondence is determined by spatial point matching between the initial registration point cloud and the global point cloud segment.

[0025] According to a second aspect of this disclosure, a three-dimensional reconstruction method is provided, the method comprising:

[0026] Based on a first image sequence of a target object acquired by a camera, a first local point cloud is generated, described by a first local reference frame; wherein, a portion of the image frames in the first image sequence are reference image frames that establish a mapping with a global point cloud segment of the target object, and the global point cloud segment is described by a global reference frame;

[0027] Based on the camera pose correspondence between the reference image frame and the global reference frame in the first local reference frame, the first local point cloud is registered at the first level to obtain the initial registered point cloud.

[0028] Based on the geometric attribute correspondence between the initial registration point cloud and the global point cloud fragment, the initial registration point cloud is registered at a second level to obtain the first global point cloud fragment of the target object described by the global reference system.

[0029] By fusing multiple global point cloud fragments of the target object described in the global reference frame, a global point cloud of the target object is obtained; wherein, the multiple global point cloud fragments include the first local point cloud;

[0030] The target object is reconstructed in three dimensions based on the global point cloud to obtain a three-dimensional model of the target object.

[0031] According to a third aspect of this disclosure, a computing device is provided, the computing device comprising:

[0032] processor;

[0033] Memory used to store processor-executable instructions;

[0034] The processor is configured to implement the method according to the first or second aspect of this disclosure when executing instructions stored in the memory.

[0035] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is also provided, having stored thereon computer program instructions that, when executed by a processor, implement the method described in the first or second aspect of this disclosure.

[0036] According to embodiments of this disclosure, local point clouds generated based on each image sequence are sequentially fused into a global map using an incremental fusion method. This incremental fusion then yields the global point cloud of the target object for 3D reconstruction. This embodiment employs a two-level registration structure for incremental point cloud fusion. The first level performs rapid coarse registration based on camera pose, while the second level performs fine registration based on point cloud geometric attributes. This registration structure avoids the high computational complexity of traditional global BA, reduces the computational resource requirements for dense point cloud generation, thereby improving 3D reconstruction efficiency and reducing reconstruction costs.

[0037] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0039] Figure 1 This is a schematic diagram of the composition structure of a data processing system capable of implementing the methods provided in the embodiments of this disclosure;

[0040] Figure 2 This is a flowchart illustrating an incremental generation method for global point clouds based on some embodiments;

[0041] Figure 3 This is a flowchart illustrating the process of generating local point clouds based on image sequences according to some embodiments;

[0042] Figure 4 This is a flowchart illustrating two-level registration according to some embodiments;

[0043] Figure 5This is a flowchart illustrating a three-dimensional reconstruction method according to some embodiments;

[0044] Figure 6 This is a schematic diagram of the composition of a computing device according to some embodiments. Detailed Implementation

[0045] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0046] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0047] Techniques, methods, and apparatus known to those skilled in the art in the relevant field may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

[0048] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0049] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0050] This disclosure relates to a scheme for 3D reconstruction of a target object based on image data. Embodiments of this disclosure generate a dense point cloud for 3D reconstruction of the target object by processing multiple consecutive image sequences of the target object. The target object in this disclosure can be a single entity such as a road or bridge, or a composite object containing multiple entities such as a scenic area. The image data in this embodiment is acquired by a camera mounted on a mobile device. This mobile device can be a general-purpose mobile device or a dedicated data acquisition device, and is not limited thereto.

[0051] The method of this disclosure embodiment can be used through Figure 1 The data processing system implementation is shown. For example... Figure 1 As shown, the data processing system includes a mobile device 101 and a computing device 102. The mobile device 101 is equipped with a camera 1015, which is used to acquire image data of the target object at a set acquisition frequency during the movement of the mobile device 101. The computing device 102 is used to process the image data acquired by the mobile device 101 to generate a global point cloud for reconstructing the target object.

[0052] like Figure 1 As shown, in addition to camera 1015, mobile device 101 may also include processor 1011, memory 1012, communication device 1013, and sensors such as wheel speed gauge 1016 and inertial measurement unit 1014 (IMU) that are capable of measuring the true displacement of mobile device 101.

[0053] Figure 1 The computing device 102 can be any physically existing device, machine, or system capable of performing predefined computing tasks through hardware components, namely, receiving input data, processing data, storing data or information, and generating output results. Examples of computing devices 102 include servers, computers, controllers, and systems-on-a-chip. The computing device 102 can be set up independently of the mobile device 101, or it can be at least partially set up within the mobile device 101; this is not limited here.

[0054] like Figure 1 As shown, the computing device 102 may include a processor 1021 and a memory 1022. The processor 1021 may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System-on-Chip (SOC), an Application Specific Integrated Circuit (ASIC), a Micro Controller Unit (MCU), or other processors. The memory 1022 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. In addition to storing instructions, the memory may also store input data, intermediate data, and result data, and the data stored in the memory can be retrieved and used by the processor.

[0055] In addition, the computing device 102 may also include a communication device 1023 and an interface device 1024, which can, for example, receive and process images acquired by the camera through the communication device 1023 or the interface device 1024.

[0056] The following is combined Figure 1 The systems shown illustrate various embodiments of this disclosure.

[0057] <First Embodiment>

[0058] Figure 2 A flowchart illustrating an incremental generation method for a global point cloud according to some embodiments is shown. This method can be... Figure 1 The computing device 102 can be implemented online or offline, and this is not limited to that. For example... Figure 2 As shown, the method may include the following steps S210 to S230.

[0059] Step S210: Based on the first image sequence of the target object acquired by the camera, generate a first local point cloud described by a first local reference frame.

[0060] The camera in this embodiment is composed of Figure 1 The mobile device 101 is equipped with a camera that continuously captures images of the target object as the mobile device 101 moves, resulting in an image set where each image frame or key image frame is timestamped. Here, the image set may include every image frame captured by the camera, or it may include key image frames captured by the camera to reduce data processing load. For example, key image frames may be selected from the images captured by the mobile device based on a setting where the translation or rotation amount between two adjacent key frames is greater than or equal to a predetermined threshold.

[0061] In this embodiment, the camera used for image acquisition can be a monocular camera, and each image frame marked by a timestamp can be a single image. In other words, the method in this embodiment is applicable to application scenarios that perform three-dimensional reconstruction of target objects based on image sets acquired by a monocular camera.

[0062] Based on the incremental point cloud generation mode of this embodiment, the image set of the target object can be split into multiple image sequences, and a global point cloud segment described by a global reference frame can be generated for each image sequence. Then, a global point cloud for three-dimensional reconstruction of the target object can be generated by fusing multiple global point cloud segments.

[0063] In this embodiment, adjacent image sequences within multiple image sequences share common image frames. This allows for point cloud registration based on previously processed image sequences when processing the current image sequence to generate the corresponding local point cloud, achieving incremental point cloud generation. In other words, each split image sequence includes multiple image frames, and some image frames from adjacent image sequences are common image frames. Different image sequences can have the same number of image frames or different numbers of image frames.

[0064] The first image sequence P in this embodiment iLet i be the image sequence generated from the current point cloud, where i is an integer greater than or equal to 2. The first image sequence P... i The forward adjacent image sequence is the second image sequence P. i-1 In the first image sequence P i When generating point clouds corresponding to the global reference frame, the second image sequence P i-1 Point cloud generation corresponding to the global reference frame has been completed.

[0065] First image sequence P i A portion of the image frames in the sequence are reference image frames that establish a mapping with a global point cloud segment of the target object described in a global reference frame. This reference image frame is also the first image sequence P. i With the second image sequence P i-1 The common image frame. The global point cloud segment mapped to this reference image frame can refer to the second image sequence P. i-1 The second global point cloud segment can also refer to the portion of the second global point cloud segment that corresponds to the reference image frame; this is not limited here.

[0066] In this embodiment, "local point cloud" refers to a point cloud defined in its own local, temporary coordinate system, also known as a "local reference system." When generating corresponding local point clouds for different image sequences, these local point clouds typically correspond to different local reference systems. For example, based on the first image sequence P... i The generated first local point cloud can be defined by a first local reference frame, based on the second image sequence P. i-1 The generated second local point cloud can be defined by a second local reference system. The first local reference system and the second local reference system can be the same or different. In this embodiment, the terms "global point cloud fragment" and "global point cloud" refer to the point cloud being defined in a unified and fixed coordinate system. All local point clouds of the target object are registered to this coordinate system, which is also known as the "global reference system".

[0067] In step S210, the computing device 102 can perform calculations based on the first image sequence P. i Geometric consistency constraints are applied between different image frames to generate a first local point cloud with scale, represented by a first local reference frame. Here, "with scale" means that the size unit of the point cloud data is consistent with real-world physical units (e.g., meters), allowing for direct physical measurement. The first local reference frame can be any reference frame established during the generation of the first local point cloud; for example, it can be based on the first image sequence P. i The first image frame P in i1 The corresponding camera coordinate system serves as the first local reference system for defining the first local point cloud.

[0068] Regarding the generation of the first local point cloud, in some examples, scale-free acquisition information related to point cloud generation can be obtained first based on the first image sequence, and then scale restoration can be performed on the scale-free acquisition information to generate the first local point cloud. Decoupling geometric reconstruction from scale can avoid the computational complexity and scene applicability limitations caused by joint estimation. In these examples, the generation of the first local point cloud described by a local reference frame based on the first image sequence of the target object acquired by the camera in step S210 can include the following steps S211-S213:

[0069] Step S211: Based on the geometric consistency constraints between different image frames in the first image sequence, the first acquisition information of the first image sequence is obtained.

[0070] The first acquired information includes camera intrinsic parameters, as well as the scale-free camera pose and scale-free pixel depth of each image frame in the first image sequence. The first acquired information is described or defined based on a first local reference frame.

[0071] The first image sequence has unified camera intrinsic parameters, which represent the camera's inherent properties such as focal length and principal point, and can be represented in pixels.

[0072] Since the rotation matrix and camera intrinsic parameters in the camera pose are scale-independent quantities and unaffected by scale ambiguity, the translation vector and pixel depth of the camera pose obtained in step S211 are scale-free values. Specifically, scale-free camera pose means that the camera's rotation matrix relative to the local reference frame has recovered its true orientation when acquiring each image frame, but the translation vector only has a relative scale. Scale-free pixel depth means that the distance from the spatial point corresponding to each pixel along the camera's optical axis to the camera only has a relative scale. Relative scale refers to the geometric quantity expressed as the proportional relationship between dimensions; its value ensures the accuracy of internal proportions but cannot directly correspond to actual physical dimensions.

[0073] Step S212: Scale restoration is performed on the acquired information of the first image sequence to obtain second acquired information with scale.

[0074] The scale recovery in step S212 refers to the scale recovery of the scale-free camera pose and scale-free pixel depth of each image frame, thereby obtaining the scaled camera pose and scaled pixel depth. Here, the second acquisition information is still described or defined based on the first local reference frame.

[0075] In step S212, the first image sequence P can be acquired by the camera based on the mobile device 102. iThe true value of the displacement is used to determine the scaling factor for scale restoration. Scale restoration can solve the scale drift problem in monocular vision reconstruction, enabling the physical size measurement of local point clouds in different image sequences.

[0076] The true displacement value can be determined based on the displacement information provided by sensors such as wheel speedometers and IMUs configured in the mobile device 102. For example, the ratio of the true displacement measured by the sensor between adjacent image frames to the relative displacement calculated by the pose of the scaleless camera can be used as the scaling factor.

[0077] The true value of the displacement can also be determined based on the scaled structure of motion (SFM) trajectory, which is the mobile device trajectory or camera trajectory obtained based on SFM. This trajectory is called a sparse trajectory because SFM processing discards image frames that do not meet the conditions.

[0078] In some examples, such as Figure 3 As shown, the first image sequence P can be... i The input is fed into an end-to-end model, which then learns the first image sequence P. i Geometric consistency regression between different image frames yields the first image sequence P. i The first collected information is obtained through a one-time regression using an end-to-end model. This eliminates the need for explicit computation, reducing computational resource requirements and improving processing efficiency while maintaining data accuracy. This end-to-end model can be, for example, the VGGT (Visual Geometry Grounded Transformer) model, or other models based on the Transformer architecture, such as a model structure combining Transformer and convolutional CNN.

[0079] In another example, the first acquisition information of the first image sequence can also be obtained by constructing a mathematical model and performing explicit calculations based on the geometric consistency constraints of different image frames.

[0080] In the example where the first collected information is obtained through a one-time regression using an end-to-end model, the scaling factor for scale recovery can be determined based on the sparse trajectory with scale in the motion reconstruction structure. This allows for compatibility with mixed inputs of local point cloud generation from the end-to-end model and SFM sparse reconstruction, overcoming the limitations of single-algorithm scenarios and generating globally consistent, scale-accurate dense point clouds in large-scale scenarios such as roads, thus providing a high-precision foundation for autonomous driving.

[0081] Step S213: Based on the second acquisition information with scale, the pixels in the first image sequence are mapped to three-dimensional space to obtain the first local point cloud.

[0082] See also Figure 3 For the first image sequence P i For each image frame in the sequence, based on the camera intrinsic parameters and the scaled camera pose and scaled pixel depth of that image frame, each pixel of the image frame is mapped into three-dimensional space through back-projection to obtain the position coordinates of the corresponding spatial point in the first local reference frame. The first image sequence P... i By stitching together the spatial points obtained by backprojecting all image frames, the first local point cloud is obtained.

[0083] Step S220: Based on the camera pose correspondence between the reference image frame and the first local reference frame and the global reference frame, perform first-level registration on the first local point cloud to obtain the initial registered point cloud.

[0084] In step S220, transformation parameters for initially transforming the first local point cloud from the first local reference frame to the global reference frame can be obtained based on the camera pose correspondence, and the first-level registration of the first local point cloud can be performed based on the transformation parameters.

[0085] The camera pose correspondence of each reference image frame can be represented by the first camera pose in the first local reference frame and the second camera pose in the global reference frame. The first and second camera poses are both scaled camera poses, which are used to determine the optical center position and optical axis direction.

[0086] The transformation parameters include rotation transformation parameters and translation transformation parameters. The rotation transformation parameters are represented by matrices, while the translation transformation parameters are represented by vectors.

[0087] In some examples, the rotation and translation transformation parameters that are related to each other can be decoupled to determine the rotation and translation transformation parameters separately. This can reduce the difficulty of solving the transformation parameters and reduce the accumulated error, so as to achieve millimeter-level trajectory alignment accuracy even in large-scale scenes or low-texture areas.

[0088] In these examples, the rotation transformation parameters for the first-level registration can be determined by decoupling the rotation relationship from the camera pose correspondence of each reference image frame. The rotation relationship corresponding to each reference image frame can be represented by the rotation deviation between the first camera pose and the second camera pose corresponding to that reference image frame.

[0089] In these examples, the translation transformation parameters for the first-level registration can be determined by correcting the translation relationship of each reference image frame in the camera pose correspondence using rotation transformation parameters. Here, the translation vector of the first camera pose of the reference image frame can be corrected using rotation transformation parameters to represent the deviation between the corrected translation vector and the translation vector of the second camera pose, thereby achieving the correction of the translation relationship in the camera pose correspondence.

[0090] In these examples, the transformation parameters for the first-level registration are obtained based on the camera pose correspondence of the reference image frame, which may further include the following steps S221 to S223:

[0091] Step S221: Based on the rotation relationship between the first camera pose and the second camera pose of each reference image frame, obtain the rotation transformation parameters.

[0092] Step S222: Apply the rotation transformation parameters to the translation vector of the first camera pose of each reference image frame to obtain the rotation-corrected translation vector of each reference image frame.

[0093] Step S223: Determine the translation transformation parameters based on the deviation between the rotation-corrected translation vector of each reference image frame and the translation vector of the second camera pose of the corresponding reference image frame.

[0094] In this example, rotation and translation are separated. Step S221 performs rotation alignment, which can be achieved by calculating the average of the rotation difference matrices of multiple reference image frames to obtain the global rotation correction amount, i.e., the rotation transformation parameter. During translation alignment, in step S222, the rotation transformation parameter is first applied to the translation vector of the first camera pose in each reference image frame to re-represent the camera optical center position in the rotation-corrected reference frame. Then, in step S223, translation optimization is performed to further adjust the optical center position based on the translation deviation, aligning it with the optical center position of the second camera pose, which serves as the reference pose.

[0095] For example, rotation alignment can include:

[0096] (1) For each reference image frame P 1j Calculate the rotation difference matrix for each pair of poses:

[0097]

[0098] in, Represents the reference image frame P 1j The rotation component corresponding to the pose of the second camera, and Represents the reference image frame P 1j Corresponding to the rotation component of the first camera pose, T represents the transpose of the matrix, through the reference image frame P. 1jCorresponding rotation difference This indicates the amount of rotation required to rotate from the first camera pose to the second camera pose.

[0099] (2) Calculate the average rotation R of the rotation matrix. align As a rotational transformation parameter:

[0100]

[0101] Where N is the number of common image frames between the first and second image sequences. In other examples, the rotation transformation parameters can also be determined by other types of averages of the rotation amounts of multiple reference image frames.

[0102] For example, translation alignment can include:

[0103] (1) The rotation transformation parameter R align Translation vector acting on the first camera pose in each reference image frame:

[0104]

[0105] in, Represents the reference image frame P 1j Translation vector in the first camera pose Represents the reference image frame P 1j The translation vector after rotation correction.

[0106] (2) Perform translation optimization to obtain the translation transformation parameters t. align :

[0107]

[0108] The minimum closed-form solution for t is:

[0109]

[0110] Step S230: Based on the geometric attribute correspondence between the initial registration point cloud and the global point cloud segment mapped by the reference image frame, perform a second-level registration on the initial registration point cloud to obtain a first global point cloud segment described by a global reference frame.

[0111] The geometric attribute correspondence can be determined based on the matching relationship between neighboring spatial points in the global point cloud segment mapped by the first-level registration point cloud and the reference image frame. This can be divided into two parts: establishing the geometric correspondence and calculating the second transformation parameters based on the geometric correspondence. When establishing the geometric correspondence, the nearest neighbor of the spatial point in the initial registration point cloud can be found in the global point cloud segment. Then, data association can be established using the spatial positions of the spatial points in the two point clouds, forming multiple spatial point pairs. After establishing the data association, the transformation parameters used for the second-level registration can be determined based on least squares, etc., so that the sum of the distances between all spatial point pairs is minimized, thus completing the second-level registration.

[0112] In this embodiment, the first global point cloud fragment is used to fuse with other global point cloud fragments of the target object to form a global point cloud of the target object, so as to perform three-dimensional reconstruction of the target object through the global point cloud.

[0113] like Figure 4 As shown, in the first local point cloud LC i During the first-level registration, the scaled camera pose LP of each image frame in the second acquisition information can also be simultaneously performed. i The first-level registration is performed to obtain the initial registered camera pose for each image frame. Then, during the second-level registration of the initial registered point cloud, the first image sequence P is simultaneously registered. i The initial registration camera pose of each image frame is used for a second-level registration to obtain the registered camera pose RP for that image frame. i Camera pose RP i This refers to the camera pose of the image frame in the global reference frame. By synchronously registering the camera pose, the image sequence P can be... i+1 As the first image sequence and the image sequence P i As the second image sequence, and for image sequence P i+1 When generating point clouds, the image sequence P is used. i+1 The central reference image frame provides the camera pose in the global reference frame.

[0114] After generating a global point cloud fragment C1 based on the first image sequence P1 of the target object, the method of this embodiment can be followed to sequentially generate corresponding global point cloud fragments based on each subsequent image sequence P1, thereby obtaining the global point cloud of the target object through an incremental generation mode. Here, when generating the corresponding global point cloud fragment C1 based on the first image sequence P1, no registration operation is required. The local point cloud generated according to step S210 corresponding to the first image sequence P1 can be used as the global point cloud fragment C1. In this case, the local reference system defining the local point cloud is used as the global reference system; alternatively, the local point cloud can be registered to a global reference system, such as a world reference system.

[0115] According to steps S210 to S230 above, the method of this embodiment of the present disclosure will sequentially fuse the local point cloud generated based on each image sequence into the global map, support the real-time continuous reconstruction of the target object, and avoid the computational burden caused by global alignment.

[0116] According to steps S210 to S230 above, this embodiment of the present disclosure employs a two-level registration structure when performing incremental fusion. The first level performs fast coarse registration based on the camera pose correspondence between common image frames in adjacent image sequences and the local and global reference frames. The second level performs fine registration based on point cloud geometric attributes to obtain a high-precision dense point cloud. This two-level registration structure avoids the high computational complexity of traditional global BA. Furthermore, fine registration based on coarse registration enables rapid convergence, thereby reducing the computational resource requirements for dense point cloud generation, improving 3D reconstruction efficiency, and reducing reconstruction costs.

[0117] <Second Embodiment>

[0118] This disclosure also provides a three-dimensional reconstruction method based on the first embodiment. For example... Figure 5 As shown, the method may include the following steps S510 to S550:

[0119] Step S510: Based on the first image sequence of the target object acquired by the camera, a first local point cloud described by a first local reference frame is generated; wherein, some image frames in the first image sequence are image frames that are mapped to a global point cloud segment of the target object, and the global point cloud segment is described by a global reference frame.

[0120] Step S520: Based on the camera pose correspondence between the reference image frame and the first local reference frame and the global reference frame, perform first-level registration on the first local point cloud to obtain the initial registered point cloud.

[0121] Step S530: Based on the geometric attribute correspondence between the initial registration point cloud and the global point cloud fragment, perform a second-level registration on the initial registration point cloud to obtain the first global point cloud fragment of the target object described by the global reference system.

[0122] Step S540: Fuse multiple global point cloud fragments of the target object described by a global reference frame to obtain the global point cloud of the target object. These multiple global point cloud fragments include a first local point cloud.

[0123] Step S550: Perform 3D reconstruction of the target object based on the global point cloud to obtain a 3D model of the target object.

[0124] This embodiment can use any reconstruction algorithm to reconstruct the global point cloud of the target object into a 3D model of the target object. For example, surface reconstruction algorithms, region growing algorithms, etc. can be used, and there is no limitation here.

[0125] In another embodiment, the 3D reconstruction of the target object can be completed gradually during the process of stitching together global point cloud fragments, without having to wait until the global point cloud is fused before performing 3D reconstruction.

[0126] The method described in this embodiment can quickly and accurately reconstruct the target object at a relatively low cost.

[0127] <Third Embodiment>

[0128] This embodiment provides a computing device, such as Figure 6 As shown, the computing device 600 includes a memory 602 and a processor 601. The memory 602 is used to store computer programs executed by the processor 601. The processor 601 is configured to implement an incremental generation method for global point clouds or a three-dimensional reconstruction method according to any embodiment of the present disclosure when executing the computer program stored in the memory 602.

[0129] The computing device may be independent of the mobile device, or at least partially integrated into the mobile device, without limitation.

[0130] In addition, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the method according to any embodiment of this disclosure.

[0131] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0132] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0133] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0134] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0135] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0137] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0139] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for incremental generation of a global point cloud, characterized in that, include: Based on a first image sequence of a target object acquired by a camera, a first local point cloud is generated, described by a first local reference frame; wherein, a portion of the image frames in the first image sequence are reference image frames that establish a mapping with a global point cloud segment of the target object, and the global point cloud segment is described by a global reference frame; Based on the camera pose correspondence between the reference image frame and the global reference frame in the first local reference frame, the first local point cloud is registered at the first level to obtain the initial registered point cloud. Based on the geometric attribute correspondence between the initial registration point cloud and the global point cloud fragment, a second-level registration is performed on the initial registration point cloud to obtain a first global point cloud fragment of the target object described by the global reference system; wherein, the first global point cloud fragment is used to merge with other global point cloud fragments of the target object to form the global point cloud of the target object.

2. The method according to claim 1, characterized in that, The rotation transformation parameters of the first-level registration are determined by decoupling the rotation relationship from the camera pose correspondence of each of the reference image frames.

3. The method according to claim 2, characterized in that, The translation transformation parameters of the first-level registration are determined by correcting the translation relationship of each reference image frame in the camera pose correspondence with the rotation transformation parameters.

4. The method according to claim 1, characterized in that, The first image sequence of the target object acquired by the camera generates a first local point cloud described by a first local reference frame, including: Based on the geometric consistency constraints between different image frames in the first image sequence, the first acquisition information of the first image sequence is obtained; wherein, the first acquisition information includes camera intrinsic parameters, as well as the scale-free camera pose and scale-free pixel depth of each image frame in the first image sequence. Scale recovery is performed on the first acquired information to obtain second acquired information with scale. Based on the second acquired information, the pixels in the first image sequence are mapped to three-dimensional space to obtain the first local point cloud.

5. The method according to claim 4, characterized in that, The first acquired information is obtained by learning the geometric consistency regression between different image frames in the first image sequence using an end-to-end model.

6. The method according to claim 4 or 5, characterized in that, The scaling factor on which the scale recovery is based is determined by the scaled sparse trajectory of the motion recovery structure.

7. The method according to any one of claims 1 to 5, characterized in that, The geometric attribute correspondence is determined by spatial point matching between the initial registration point cloud and the global point cloud fragment.

8. A three-dimensional reconstruction method, characterized in that, Based on a first image sequence of a target object acquired by a camera, a first local point cloud is generated, described by a first local reference frame; wherein, a portion of the image frames in the first image sequence are reference image frames that establish a mapping with a global point cloud segment of the target object, and the global point cloud segment is described by a global reference frame; Based on the camera pose correspondence between the reference image frame and the global reference frame in the first local reference frame, the first local point cloud is registered at the first level to obtain the initial registered point cloud. Based on the geometric attribute correspondence between the initial registration point cloud and the global point cloud fragment, the initial registration point cloud is registered at a second level to obtain the first global point cloud fragment of the target object described by the global reference system. By fusing multiple global point cloud fragments of the target object described in the global reference frame, a global point cloud of the target object is obtained; wherein, the multiple global point cloud fragments include the first local point cloud; The target object is reconstructed in three dimensions based on the global point cloud to obtain a three-dimensional model of the target object.

9. A computing device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 8 when executing instructions stored in the memory.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images