Dynamic scene high-precision three-dimensional reconstruction method based on inter-frame geometric constraint and motion tracking

Abstract

The invention discloses a dynamic scene high-precision three-dimensional reconstruction method based on inter-frame geometric constraint and motion tracking. The method comprises the steps of system calibration and data acquisition; wrapping phase calculation; performing inter-frame geometric constraint phase unwrapping based on motion tracking; performing phase unwrapping correction based on confidence coefficient driving; reconstructing a three-dimensional coordinate; and updating and iterating the information. According to the method, reliable three-dimensional information of a historical frame is transmitted to a current frame through motion tracking, a dynamic self-adaptive reference is provided for geometric constraint unwrapping, the problem of pixel misalignment caused by motion is fundamentally solved, and the method is suitable for dynamic scene three-dimensional measurement of continuous motion; and in combination with inter-frame geometric constraint and motion tracking, the accuracy of phase unwrapping is improved. The introduced confidence coefficient driven correction mechanism can effectively recognize and repair abnormal points generated by motion estimation errors, shielding, depth jump and the like, and the anti-interference capability and the overall reconstruction precision of the algorithm in a complex scene are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of optical 3D measurement and computer vision, specifically relating to a high-precision 3D reconstruction method for dynamic scenes based on inter-frame geometric constraints and motion tracking. Background Technology

[0002] Optical 3D measurement technology has been widely applied in fields such as industrial inspection, intelligent manufacturing, reverse engineering, medical diagnosis, and cultural heritage protection. Among them, structured light 3D measurement technology based on stripe coding, especially fringe projection profilometry (FPP), has become a current research and application hotspot due to its advantages such as non-contact operation, high precision, high efficiency, and relatively simple system structure.

[0003] In FPP (Focus-Proof) technology, to acquire the 3D information of an object, an coded fringe pattern needs to be projected onto the object's surface, and the camera captures the deformed fringes modulated by the object's shape. By calculating the phase information of the deformed fringes, a correspondence between the camera and projector pixels is established, and then the object's 3D coordinates are recovered based on the principle of triangulation. However, the phase directly calculated using methods such as phase-shifting is wrapped in... The wrapped phase within the interval is periodic and cannot be directly used for unique matching. Therefore, phase unwrapping is required to obtain absolute, continuous phase values, i.e., absolute phase.

[0004] Traditional phase unwrapping methods are mainly divided into temporal phase unwrapping and spatial phase unwrapping. Temporal phase unwrapping (such as multi-frequency heterodyne and Gray code methods) requires projecting multiple coded patterns, which can obtain high-precision results in static scenes. However, in dynamic scenes, the motion of objects causes the multiple projected patterns to become misaligned at the pixel level, introducing serious motion errors and leading to unwrapping failure. Although spatial phase unwrapping only requires a single phase map, it relies on the assumption of smooth surface and has poor robustness in the presence of depth jumps, isolated objects, or noise interference, making it prone to error propagation.

[0005] To address the challenges of dynamic scenes, existing research has attempted to reduce the number of projection patterns (e.g., composite stripes, color coding), introduce additional hardware (e.g., multi-view systems), or utilize prior information (e.g., CAD models). However, these methods often suffer from high hardware costs, system complexity, limited applicability, or weak anti-interference capabilities. In particular, for standard single-camera-single-projector FPP systems, achieving reliable and high-precision phase unfolding and 3D reconstruction in moving scenes without projecting additional unwrapping patterns or increasing hardware complexity remains a pressing technical challenge. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing technologies and provide a high-precision 3D reconstruction method for dynamic scenes based on inter-frame geometric constraints and motion tracking. It combines a confidence-driven mechanism to optimize and correct the unwrapping results, thereby achieving fast and accurate 3D reconstruction of moving objects without the need for additional projection unwrapping patterns and hardware support.

[0007] The technical solution for achieving the objective of this invention is: a high-precision 3D reconstruction method for dynamic scenes based on inter-frame geometric constraints and motion tracking, comprising:

[0008] Step 1: Build an FPP system consisting of a camera and a projector, complete system calibration, and obtain the internal and external parameters of the camera and projector; control the projector to continuously project a set of phase-shifted fringe patterns onto the moving object, and the camera synchronously triggers the acquisition of the deformed fringe image sequence modulated by the object;

[0009] Step 2: For the acquired single-set phase-shifted fringe images, calculate the wrapping phase map of the current frame using the phase-shifting method;

[0010] Step 3: Motion tracking is performed based on the motion continuity between adjacent frames to provide a reliable geometric constraint reference for the current frame, thereby solving for the absolute phase;

[0011] Step 4: For the unreliable absolute phase results in Step 3, perform phase unwrapping correction based on confidence evaluation to obtain the optimized absolute phase map of the current frame;

[0012] Step 5: Using the absolute phase map of the current frame and the system calibration parameters, calculate the three-dimensional spatial coordinates of each point on the object surface through the principle of triangulation to complete the three-dimensional reconstruction of the current frame;

[0013] Step 6: Store and update the 3D point cloud and corresponding absolute phase information reconstructed from the current frame, and use it as the reference information of the previous frame for processing the next frame image, thereby realizing real-time 3D reconstruction in continuous dynamic scenes.

[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0015] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described method.

[0016] A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] (1) Strong dynamic adaptability: By transmitting reliable 3D information from historical frames to the current frame through motion tracking, it provides a dynamic adaptive reference for unwrapping geometric constraints, fundamentally solving the pixel misalignment problem caused by motion, and is suitable for 3D measurement of dynamic scenes with continuous motion.

[0019] (2) Low hardware requirements and simple system: No need to project additional unpacking encoding patterns (such as Gray code), nor to add additional hardware devices such as cameras or projectors. Only a standard monocular FPP system is needed, which reduces system cost and complexity and expands the field of view and applicable scenarios.

[0020] (3) High accuracy and robustness: Combining inter-frame geometric constraints and motion tracking improves the accuracy of phase unfolding. The introduced confidence-driven correction mechanism can effectively identify and repair abnormal points caused by motion estimation errors, occlusion, depth jumps, etc., significantly improving the algorithm's anti-interference ability and overall reconstruction accuracy in complex scenes.

[0021] (4) High potential for real-time performance: This method avoids the projection and acquisition time of multiple coded patterns, requires fewer images for a single measurement, and combined with an efficient motion estimation algorithm, it is conducive to realizing real-time 3D reconstruction in high-speed dynamic scenes. Attached Figure Description

[0022] Figure 1 This is a flowchart of the rapid three-dimensional measurement technology for motion scenarios in step three of the present invention.

[0023] Figure 2 This is a flowchart of the confidence-driven phase expansion algorithm in step four of this invention.

[0024] Figure 3 The results of 3D reconstruction using different schemes are: (a) using motion tracking scheme; (b) not using motion tracking scheme.

[0025] Figure 4 This is a schematic diagram illustrating the fitting accuracy of a standard sphere and a standard plane.

[0026] Figure 5 Here are schematic diagrams of the three-dimensional values ​​and confidence levels of the geometric constraints: (a) Three-dimensional reconstruction results; (b) Confidence level distribution.

[0027] Figure 6 These are the 3D reconstruction results of a deeply moving object: (a) First frame; (b) Second frame; (c) Third frame. Detailed Implementation

[0028] This invention proposes a high-precision 3D reconstruction method for dynamic scenes based on inter-frame geometric constraints and motion tracking, comprising the following steps:

[0029] Step 1: System Calibration and Data Acquisition. Build an FPP system consisting of a single camera and a single projector, complete system calibration, and acquire the intrinsic and extrinsic parameters of the camera and projector. Control the projector to continuously project a set of phase-shifted fringe patterns onto the moving object, while the camera synchronously triggers the acquisition of the deformed fringe image sequence modulated by the object.

[0030] Step 2: Wrapping Phase Calculation. For the acquired single-set phase-shifted fringe images (such as three-step or four-step phase shifts), the wrapping phase map of the current frame is calculated using the phase-shifting method.

[0031] Step 3: Inter-frame geometric constraint phase unrolling based on motion tracking. This step is the core of the invention, aiming to utilize the motion continuity between adjacent frames to provide a reliable geometric constraint reference for the current frame, thereby solving for the absolute phase. Specifically, it includes:

[0032] 1) Motion estimation and tracking: Using the reconstructed 3D point cloud information from the previous frame (as an initial reference), combined with the wrap phase of the current frame and the geometric constraints of the system, the rigid body motion of the object from the previous frame to the current frame (rotation matrix R and translation vector t) is estimated, and the motion matrix from the previous frame to the current frame is obtained without relying on feature point extraction.

[0033] 2) Reference value generation: Using the estimated motion matrix, the absolute phase (or the corresponding 3D point) of the previous frame is transformed to the view of the current frame, generating a predicted absolute phase reference value for each pixel (or its neighborhood) in the current frame.

[0034] 3) Absolute Phase Calculation: For each pixel in the current frame, its wrapping phase is compared with the absolute phase reference value predicted in step 2. According to the geometric constraint principle, the absolute phase differs from the wrapping phase. Multiples of . By finding the integer multiples of the difference between the two. The stripe level of a pixel can be determined by multiples of the interval. Then the absolute phase is calculated. ,in For the wrapping phase.

[0035] Step 4: Confidence-Driven Phase Unwrapping Correction. To address the unreliable unwrapping results in Step 3 that may be caused by motion estimation errors, depth jumps, newly appearing regions, or noise, a confidence evaluation and correction mechanism is introduced:

[0036] 1) Confidence score calculation: Based on the degree of agreement between the predicted absolute phase reference value and the final selected solution, the consistency of the neighborhood phase, and the continuity of the phase between frames when solving the absolute phase in step 3, a confidence score is assigned to the unwrapping result of each pixel.

[0037] 2) Error identification and isolation: Pixels with confidence levels below a set threshold are identified as unreliable points or potential error points and isolated to prevent their errors from affecting other reliable areas.

[0038] 3) Phase correction and filling: For isolated low-confidence regions, phase correction and filling are performed using the phase information of high-confidence points in their spatial neighborhood (through spatial smoothing or interpolation), or using the information of high-confidence regions in the preceding / following frames in the time series, to finally obtain a complete and highly reliable absolute phase map.

[0039] Step 5: 3D Coordinate Reconstruction. Using the absolute phase map of the current frame obtained in Step 3 (or after optimization in Step 4), combined with the system calibration parameters, the 3D spatial coordinates of each point on the object surface are calculated using the principle of triangulation, thus completing the 3D reconstruction of the current frame.

[0040] Step Six: Information Update and Iteration. The 3D point cloud and corresponding absolute phase information reconstructed from the current frame are stored and updated as "previous frame" reference information for processing the next frame, thereby achieving real-time 3D reconstruction in continuous dynamic scenes.

[0041] Furthermore, the motion estimation in step three employs an Iterative ClosestPoint (ICP) optimization method, which optimizes the solution of the motion matrix by minimizing the distance between the initial 3D point set generated by the wrapping phase and geometric constraints in the current frame and the corresponding points of the 3D point set in the previous frame after motion transformation.

[0042] Furthermore, in step three, when solving for the absolute phase using geometric constraints, a threshold for the allowable depth variation range based on system parameters is set (e.g., 30mm). When the difference between the predicted absolute phase reference value and the three-dimensional depth calculated from a candidate absolute phase is within this threshold, the candidate solution is considered reasonable.

[0043] Furthermore, the confidence calculation in step four takes into account one or more of the following factors: the magnitude of the residual between the current pixel unwrapping result and the predicted absolute phase reference value; the consistency between the current pixel unwrapping result and the unwrapping results of confirmed high-confidence pixels in its spatial neighborhood; and the stability of the unwrapping results in the region where the current pixel is located over multiple consecutive frames.

[0044] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0045] Example

[0046] This embodiment provides a high-precision 3D reconstruction method for dynamic scenes based on inter-frame geometric constraints. The specific steps are as follows:

[0047] Step 1: System Setup and Initialization. A monocular FPP system is constructed using a DLP projector and a CMOS industrial camera, arranged in a typical intersecting configuration. High-precision calibration of the camera and projector is achieved using the Zhang Zhengyou calibration method combined with phase assistance, obtaining the intrinsic parameter matrix, distortion coefficients, and the rotation and translation relationship between them. At the start of measurement, a high-precision absolute phase map and 3D point cloud of the initial static scene (or the starting frame of motion) are obtained using a traditional multi-frame encoded temporal phase unfolding method (such as four-step phase shift combined with four Gray codes). This serves as the initial reference information for the "previous frame" in subsequent frame processing.

[0048] Step Two: Dynamic Sequence Acquisition and Wrap-up Phase Acquisition. The projector is controlled to cyclically project a set of N-step (e.g., N=3 or 4) phase-shifted sinusoidal fringe patterns at a fixed frequency (e.g., 60Hz). The pattern captured by the camera can be represented as:

[0049]

[0050] in, This indicates a point on the image plane. The light intensity distribution measured at that location, Represents camera pixel coordinates, Indicates a striped background. Indicates adjustment system, This indicates that the phase needs to be obtained.

[0051] The camera and projector are triggered synchronously to acquire images of deformed stripes modulated by the moving object. For each acquired phase-shift image, the wrap-around phase image at that moment (frame) is calculated in real time using arctangent calculation. .

[0052] Step 3: Fast inter-frame 3D measurement based on motion tracking, such as... Figure 1 As shown.

[0053] Preliminary 3D point generation: Wrapping phase of the current frame Using the system's geometric constraints, an initial fringe order is assumed for the wrapping phase of each pixel (e.g., assuming...). A corresponding preliminary three-dimensional point set can be calculated. Because the stripe order may be incorrect, It is a discontinuous 3D point cloud with "folds".

[0054] Motion matrix estimation: This involves reconstructing the correct 3D point cloud from the previous frame. As a reference model, the preliminary 3D point set of the current frame As the data to be matched, the ICP algorithm based on nearest neighbor search is used to iteratively solve for a rigid body motion transformation matrix. , so that the transformed and The corresponding points are at the minimum distance. The transformation matrix here... This is the estimated motion matrix from frame t-1 to frame t. To accelerate and robustly estimate the motion, the Random Sample Consensus (RANSAC) approach or optimization using small-angle motion priors can be employed.

[0055] Reference absolute phase generation: using the motion matrix obtained in the previous step The absolute phase map of the previous frame Or directly use its 3D point cloud Motion compensation is performed, and the image is projected onto the camera's viewpoint of the current frame to obtain the predicted absolute phase reference map for the current frame. .

[0056] Absolute phase solution: for each pixel in the current frame Its encapsulation phase is Its absolute phase should satisfy:

[0057]

[0058] Meanwhile, the absolute phase reference value predicted at this point is According to geometric constraints, the correct fringe order. The calculated absolute phase should be such that Compared with the predicted absolute phase reference value The difference As small as possible, and theoretically, in regions where the object moves continuously and the depth change is minimal. It should be close to 0. Therefore, by searching within a finite range (such as...) integers The choice makes The smallest one This allows us to determine the absolute phase of the pixel. .

[0059] Step 4: Confidence-driven correction, such as... Figure 2 As shown, it is suitable for occasions with extremely high accuracy requirements or complex scenarios.

[0060] 1) Calculate the initial confidence level: For the absolute phase of each pixel obtained in step 3 and its corresponding optimal Calculate its absolute phase reference value compared to the prediction. residual Residual The smaller the value, the higher the confidence level. Simultaneously, the gradient of the unwrapped absolute phase of the pixel compared to the absolute phase of pixels in its 8-neighborhood is examined. If the gradient is smooth, the spatial consistency confidence level is increased.

[0061] 2) Divide into reliable and unreliable regions: Set a confidence threshold. . (The following is a list of terms:) ...confidence level higher than or equal to Pixels marked as "reliable points" have their absolute phase directly adopted. Pixels with confidence levels below a threshold are considered "reliable points". The pixel is marked as "unreliable point" and its phase value is temporarily set to invalid.

[0062] 3) Phase repair in unreliable regions:

[0063] Spatial restoration: For small, isolated unreliable regions, the absolute phase of the region is restored by using the absolute phase of the surrounding reliable points through bilinear interpolation or a region-filling method based on the Poisson equation.

[0064] Timing restoration: For regions that lack a reference from the previous frame due to the new appearance of an object's movement (such as the back of an object exposed by rotation), or large unreliable regions, their wrapping phase can be temporarily retained. After processing the subsequent frames, when the region becomes a reliable region in another frame, the absolute phase of the current frame can be restored by using the motion relationship between multiple frames.

[0065] 4) Confidence Transfer and Update: Store the high-confidence absolute phase map and corresponding 3D point cloud obtained after repairing the current frame, and update the confidence labels of each point for processing in the next frame. For previously unreliable points that have been successfully repaired, their confidence can be appropriately increased.

[0066] Step 5: 3D Reconstruction. Utilizing the absolute phase map of the current frame determined in Steps 3 and 4. Based on the camera-projector pixel correspondence model established by the system calibration, the three-dimensional coordinates of each pixel in the world coordinate system are calculated using triangulation formulas. Generate a complete 3D point cloud model for the current frame.

[0067] Step Six: Iterate through the loop. Use the reconstructed 3D point cloud, absolute phase map, and updated confidence information of the current frame as the new "previous frame" data. Repeat steps two through five to achieve frame-by-frame 3D reconstruction of a continuous dynamic scene.

[0068] To verify the effectiveness of the method of the present invention, multiple sets of experiments were conducted on a constructed monocular FPP system. The experimental subjects included regular objects undergoing translational and rotational motion. Figure 3 The results of 3D reconstruction are shown with and without motion tracking. Figure 4This paper demonstrates the results of three-dimensional reconstruction and fitting of the surface morphology of a standard plate and a standard sphere under motion conditions using the method of this invention. Experimental results show that the fitting accuracy of the three-dimensional morphology is within 0.06 mm, and the reconstruction accuracy remains at a high level. Figure 5 A schematic diagram showing the three-dimensional values ​​and confidence levels of the geometric constraints is presented. Figure 6 The invention demonstrates that the method can stably output high-quality 3D reconstruction sequences during continuous motion of objects at great depths, proving its good dynamic adaptability and robustness.

[0069] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention.

Claims

1. A high-precision 3D reconstruction method for dynamic scenes based on inter-frame geometric constraints and motion tracking, characterized in that, include: Step 1: Build an FPP system consisting of a camera and a projector, complete system calibration, and obtain the internal and external parameters of the camera and projector; control the projector to continuously project a set of phase-shift fringe patterns onto the moving object, and the camera synchronously triggers the acquisition of the phase-shift fringe pattern sequence that is modulated and deformed by the object; Step 2: For the acquired single-set phase-shifted fringe pattern, calculate the wrapping phase map of the current frame using the phase-shifting method; Step 3: Motion tracking is performed based on the motion continuity between adjacent frames to provide a reliable geometric constraint reference for the current frame, thereby solving for the absolute phase; Step 4: For the unreliable absolute phase results in Step 3, perform phase unwrapping correction based on confidence evaluation to obtain the optimized absolute phase map of the current frame; Step 5: Using the absolute phase map of the current frame and the system calibration parameters, calculate the three-dimensional spatial coordinates of each point on the object surface through the principle of triangulation to complete the three-dimensional reconstruction of the current frame; Step 6: Store and update the 3D point cloud and corresponding absolute phase information reconstructed from the current frame, and use it as the reference information of the previous frame for processing the next frame image, thereby realizing real-time 3D reconstruction in continuous dynamic scenes.

2. The method for high-precision 3D reconstruction of dynamic scenes based on inter-frame geometric constraints and motion tracking according to claim 1, characterized in that, Step 3: Motion tracking is performed based on the motion continuity between adjacent frames to provide a reliable geometric constraint reference for the current frame, thereby solving for the absolute phase. This specifically includes: 1) Using the reconstructed 3D point cloud information from the previous frame, combined with the wrapping phase and system geometric constraints of the current frame, estimate the rigid body motion of the object from the previous frame to the current frame, and obtain the motion matrix from the previous frame to the current frame. 2) Using the estimated motion matrix, the absolute phase of the previous frame is transformed to the viewpoint of the current frame, generating a predicted absolute phase reference value for each pixel in the current frame; 3) For each pixel in the current frame, compare its wrapped phase with the absolute phase reference value predicted in step 2); according to the geometric constraint principle, the absolute phase and the wrapped phase differ. Multiples of integers; by finding the difference between the two is within a certain range. Integer multiples within the interval determine the stripe level of the pixel. Then the absolute phase is calculated. ,in For the wrapping phase.

3. The method for high-precision 3D reconstruction of dynamic scenes based on inter-frame geometric constraints and motion tracking according to claim 2, characterized in that, In step three, a nearest neighbor-based iterative optimization method is used to optimize the solution of the motion matrix by minimizing the distance between the initial three-dimensional point set generated by the wrapping phase and geometric constraints in the current frame and the corresponding points of the three-dimensional point set in the previous frame after motion transformation.

4. The method for high-precision 3D reconstruction of dynamic scenes based on inter-frame geometric constraints and motion tracking according to claim 2, characterized in that, In step three, when solving for the absolute phase using geometric constraints, a threshold for the allowable depth variation range based on system parameters is set. When the difference between the predicted absolute phase reference value and the three-dimensional depth calculated from a candidate absolute phase is within this threshold, the candidate solution is considered reasonable.

5. The method for high-precision 3D reconstruction of dynamic scenes based on inter-frame geometric constraints and motion tracking according to claim 1, characterized in that, Step four: For the unreliable absolute phase results in step three, phase unwrapping correction is performed based on confidence evaluation to obtain the optimized absolute phase map of the current frame, specifically including: 1) Based on the degree of agreement between the predicted absolute phase reference value and the final selected solution, the consistency of the neighborhood phase, and the continuity of the inter-frame phase information when solving the absolute phase in step 3, assign a confidence score to the unwrapping result of each pixel. 2) Pixels with a confidence level below a set threshold are identified as unreliable points or potential error points and isolated to prevent their errors from affecting other reliable areas; 3) For isolated low-confidence regions, phase correction and filling are performed using the phase information of high-confidence points in their spatial neighborhood, or using the information of high-confidence regions in the previous frame in terms of time, to obtain an absolute phase map.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-5.

Images