A method for real-time 3D reconstruction based on dynamic objects in video

Abstract

The present invention discloses a method for real-time 3D reconstruction based on dynamic targets in a video, comprising the following steps: S1, capturing a video sequence and extracting image frames and corresponding depth images; S2, performing target extraction and depth processing to generate an initial facet set; S3, constructing an inter-frame registration graph structure based on SurfelWarp and performing non-rigid registration; S4, performing facet activation detection based on depth changes and boundary features and inserting new facets; S5, determining the spatial distance and normal angle between facets and performing facet merging; S6, counting the number of registrations and occlusion status, and eliminating facets that do not meet the survival condition; S7, adjusting the facet density based on the registration error; S8, outputting the updated facet set and registration graph structure to generate a 3D reconstruction model sequence. The present invention realizes real-time 3D modeling for dynamic targets.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a method for real-time three-dimensional reconstruction based on dynamic targets in a video. Background Art

[0002] With the continuous development of computer vision, graphics, and artificial intelligence, 3D reconstruction technology has been widely used in many key fields, including augmented reality, autonomous driving, robot navigation, medical image analysis, and digital human modeling. The goal of 3D reconstruction is to infer the geometric shape, surface properties, and spatial structure of objects in a scene through multi-source perception information. Among many reconstruction tasks, real-time 3D reconstruction based on video streams has gradually become a research hotspot due to its advantages such as high information density, strong temporal continuity, and low deployment cost. However, there are a large number of dynamic targets in videos. These targets often have characteristics such as non-rigid deformation, self-occlusion, and rapid movement. Traditional 3D modeling algorithms are difficult to directly apply to such dynamic scenes.

[0003] Most existing 3D reconstruction methods based on image sequences rely on technical frameworks such as structured light, binocular vision, SfM, and SLAM. The core idea of ​​these methods is usually based on the rigidity assumption between multiple frames, that is, it is assumed that the objects in the scene maintain the same geometric shape between consecutive frames. Based on this assumption, by extracting image feature points and performing inter-frame matching, the camera pose can be estimated and a dense point cloud can be restored. However, when the video contains dynamic objects such as human bodies, animals, and robotic arms, the rigidity assumption is difficult to hold, resulting in matching errors, structural distortion, and reconstruction failure. In addition, traditional dense reconstruction algorithms rely on voxel or point cloud representation, which has poor spatial continuity, high memory consumption, and difficulty in dynamic structural adjustment. They are not suitable for real-time applications with high frame rates and low latency.

[0004] In recent years, a series of non-rigid 3D reconstruction methods have been proposed to address the need for 3D modeling of dynamic objects. Among these, algorithms such as SurfelWarp, DynamicFusion, and FWD-SLAM attempt to use the surfel structure as the fundamental unit of scene modeling. They convert each depth map frame into a collection of surfels and achieve structural alignment through inter-frame registration. Compared to voxel and point cloud representations, these methods offer higher spatial efficiency and structural representation capabilities, and are more resilient in handling non-rigid deformations. Surfels typically contain information such as position, normal, radius, color, and confidence. They can be constructed into structural frames in continuous time, forming a time-evolving 3D model. However, existing methods still have limitations in their non-rigid registration mechanisms. Most algorithms restrict the registration process to a local rigid transformation, performing interpolation via control points or graph nodes. However, these algorithms lack efficient constraints on the interpolated structure, making it difficult to balance local flexibility with global stability. Furthermore, registration errors are typically used only for rough cropping or mismatch elimination, lacking structural adaptive capabilities based on error inverse regulation. In addition, existing studies often ignore the continuity and state evolution laws of surface element attributes in the time dimension, and fail to build a stable and high-quality reconstruction model sequence.

[0005] To improve the robustness and real-time performance of non-rigid 3D reconstruction in dynamic scenes, key issues include: how to establish a stable surface registration graph structure between frames, how to construct an efficient and optimizable non-rigid transformation function, how to use errors to drive dynamic adjustment of surface density, and how to perform continuous time updates and structural output on surface sets. In traditional methods, the registration graph structure is usually constructed statically, making it difficult to cope with changes in factors such as occlusion, disappearance, and deformation. The non-rigid transformation function is simple in form and cannot provide flexible deformation under the constraints of the graph structure. Error evaluation results are often only used as auxiliary information rather than participating in the main modeling process. The surface set is usually used as a static or low-update frequency data structure, lacking a frame-level dynamic control mechanism. These problems lead to existing methods suffering from defects such as reduced 3D reconstruction accuracy, increased time delay, and local structural distortion when faced with video scenes with rapid motion, frequent occlusion, or drastic deformation.

[0006] Therefore, how to provide a method for real-time 3D reconstruction based on dynamic objects in a video is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0007] One purpose of the present invention is to propose a method for real-time three-dimensional reconstruction based on dynamic targets in videos. The present invention makes full use of surface element-level dynamic modeling, graph structure-controlled non-rigid registration mechanism and error-driven density adjustment strategy, and describes in detail the entire process of inter-frame registration graph construction, local rigid interpolation transformation modeling, surface element set update and reconstruction model sequence output for dynamic objects in video sequences. It has the advantages of flexible structural expression, strong temporal consistency and strong ability to adapt to dynamic deformation.

[0008] According to an embodiment of the present invention, a method for real-time 3D reconstruction based on a dynamic target in a video includes the following steps:

[0009] S1. Collect video sequences and extract each image frame and the corresponding depth image;

[0010] S2. Perform target region extraction and depth image processing on each image frame to generate an initial facet set;

[0011] S3, constructing an inter-frame registration graph structure based on the SurfelWarp facet modeling structure, performing non-rigid registration using the facet sets of the current image frame and the previous image frame, and updating the facet set of the current image frame;

[0012] S4. Performing surface element activation detection based on the depth change rate and boundary features of the current image frame to determine whether to insert a new surface element, and adding it to the surface element set of the current image frame if the conditions are met;

[0013] S5. In the facet set of the current image frame, if the spatial distance and normal angle between any two facets are both less than a set threshold, they are merged into a new facet;

[0014] S6. Count the number of registrations and occlusion status of each facet in the facet set of the current image frame in consecutive image frames, and delete facets that do not meet the survival condition.

[0015] S7, calculating a reconstruction error map based on the registration results of the current image frame and the previous image frame, and adjusting the bin density of each area of ​​the current image frame according to the reconstruction error map;

[0016] S8. Output the facet set and inter-frame registration graph structure of the current image frame, complete the frame-by-frame 3D reconstruction of the dynamic target in the video, and generate a 3D reconstruction model sequence.

[0017] Optionally, the face element includes a three-dimensional spatial position, a normal direction, a radius value, a color vector, and an active state label.

[0018] Optionally, the S1 specifically includes:

[0019] S11, decoding the video sequence and extracting continuous image frames;

[0020] S12, performing camera intrinsic calibration on each image frame, and obtaining a depth image corresponding to the image frame according to the calibration result;

[0021] S13 , performing time synchronization and pixel-level registration on the image frame and the corresponding depth image to form a one-to-one corresponding image frame and depth image combination.

[0022] Optionally, the S2 specifically includes:

[0023] S21, performing image enhancement processing based on the grayscale histogram on each image frame to improve the contrast of the edge area of ​​the image;

[0024] S22, performing edge detection based on a gradient operator based on the enhanced image frame to extract the boundary contour of the target area;

[0025] S23, constructing a target area mask in the image frame according to the boundary contour extraction result;

[0026] S24, performing noise filtering and hole filling processing on each depth image to generate a structurally continuous depth image;

[0027] S25, mapping the target area mask to the depth image, and extracting a set of depth values ​​corresponding to the target area;

[0028] S26, calculating a three-dimensional coordinate point according to each pixel position and corresponding depth value in the target area;

[0029] S27: construct a set of all three-dimensional coordinate points to generate an initial facet set corresponding to the current image frame.

[0030] Optionally, the S3 specifically includes:

[0031] S31, defining the surfel sets of the current image frame and the previous image frame as the current surfel set and the reference surfel set respectively;

[0032] S32, constructing an inter-frame registration graph structure with each surfel in the current surfel set as a node, and constructing an edge set based on the spatial adjacency relationship between the surfels;

[0033] S33, establishing a one-to-one correspondence between the current surfel set and the reference surfel set based on the coordinate mapping relationship and depth information between the image frames;

[0034] S34, let the three-dimensional position of the i-th facet in the current image frame be , the corresponding reference bin position in the previous image frame is , define the objective function of non-rigid registration:

[0035] ;

[0036] Among them, E is the objective function of non-rigid registration, is the three-dimensional coordinate of the i-th surface element in the current image frame, is the three-dimensional coordinate of the jth surface element in the current image frame, is the corresponding reference bin coordinate in the previous image frame, is the spatial transformation function to be optimized, n is the number of facets in the current image frame, is the set of all edges in the registration graph structure, Indicates the two facet indices that are connected. is the smoothing term weight coefficient, Represents the three-dimensional Euclidean norm, which is used to measure the distance between spatial points;

[0037] S35, the spatial transformation function The structure is:

[0038] ;

[0039] in, To act on the surface element The non-rigid spatial transformation result is For the face element The adjacent graph structure node index set, For the surface element The weight coefficient of node k satisfies the requirement that the sum of all weights is 1. is the three-dimensional rotation matrix corresponding to node k, is the three-dimensional translation vector corresponding to node k, in the expression Represents the node k the imposed local rigid transformation;

[0040] S36, performing iterative optimization based on gradient descent on the objective function E to obtain all variable parameters in the spatial transformation function T;

[0041] S37. For each face element in the current image frame, use the solved spatial transformation function T to update the three-dimensional spatial position, and update the normal direction according to the rotation matrix in the transformation function, while keeping the radius value and color vector unchanged. Combine the updated three-dimensional spatial position, normal direction, radius value, color vector and the original active state label to form a new face element attribute, and generate all transformed face elements in sequence to form a face element set after registration.

[0042] Optionally, the S4 specifically includes:

[0043] S41, performing depth gradient calculation on each pixel position in the current image frame to obtain a depth change rate of each pixel position;

[0044] S42, constructing a three-dimensional spatial mask based on the initial facet set of the current image frame, for identifying all spatial locations represented by the facets;

[0045] S43, traversing all pixel positions in the image frame, determining whether each pixel position is in an area not represented by a bin in the three-dimensional space, and marking pixel positions that meet the uncovered condition as candidate new bin positions;

[0046] S44. Calculate the boundary feature density for all candidate newly added bin positions, and perform bin activation judgment based on the judgment conditions:

[0047] ;

[0048] in, Represents the pixel position in the image frame The depth change rate, Indicates pixel position The density of boundary features near represents the threshold value of the depth change rate, represents the threshold value of boundary feature density, x represents the column index of the pixel, and y represents the row index of the pixel;

[0049] S45. For pixel positions that meet the activation conditions, perform a bin initialization operation to generate three-dimensional coordinates, normal direction, radius, and color value, and form a newly added bin;

[0050] S46: Merge all newly added bins into the bin set of the current image frame to form an updated bin set.

[0051] Optionally, the S5 specifically includes:

[0052] S51. In the facet set of the current image frame, traverse all facet pairs and extract the three-dimensional position and normal direction of each facet pair;

[0053] S52. For each facet, respectively calculate the angle between the spatial Euclidean distance between the three-dimensional positions and the normal direction;

[0054] S53. Assume that the position vectors of the two surface elements are and , the normal direction vectors are and , define the panel merging criterion:

[0055] ;

[0056] in, and represents the three-dimensional coordinates of two surface elements, and represents the corresponding unit normal vector, represents the three-dimensional Euclidean distance between two surface elements, represents the dot product of two normal vectors, Indicates the angle in radians to the normal direction. is the preset spatial distance threshold, is the preset normal angle threshold;

[0057] S54. For all facet pairs that meet both the spatial distance and normal angle thresholds, perform a facet merging operation, perform weighted fusion of the geometry and color attributes of the two facets, and generate a new merged facet.

[0058] S55: Replace the original bins with all the merged bins to form an updated bin set of the current image frame.

[0059] Optionally, the S6 specifically includes:

[0060] S61. Create a registration record list for each facet in the facet set of the current image frame, recording the number of times each facet participates in registration in consecutive image frames and the number of times each facet is marked as occluded in consecutive image frames;

[0061] S62. For each facet, count the total number of registration participations and the total number of occlusion marks;

[0062] S63: Setting a minimum allowable number of registration times and a maximum allowable number of occlusion times. When the number of registration times of a facet is less than the minimum allowable value, or the number of occlusion times of a facet is greater than the maximum allowable value, the facet is determined to not meet the retention requirement.

[0063] S64, deleting the facets that do not meet the retention requirement from the facet set of the current image frame;

[0064] S65: Retain the bins that meet the retention requirement and form an updated bin set of the current image frame.

[0065] Optionally, the S7 specifically includes:

[0066] S71. Calculate a 3D registration error value for each facet in the facet set of the current image frame, and construct a reconstruction error map.

[0067] S72, dividing the image frame into a set of spatial grid regions, and counting the number of facets and the average error value for each spatial grid region;

[0068] S73. Let the average error value of each spatial grid area be , the current number of pixels is d, and the error threshold is , the lower limit of the surface element density is , the upper limit of the surface element density is , define the surface element density adjustment function:

[0069] ;

[0070] in, represents the number of facets in the spatial grid area that should be adjusted, is the increase factor when the error is too high, is the reduction factor when the error is too low;

[0071] S74. For the spatial grid area that meets the addition conditions, interpolation sampling is performed based on the depth map and boundary features in the current image frame to generate the three-dimensional position and normal direction of the newly added surface element;

[0072] S75. For the spatial grid area that meets the reduction condition, select the facets with the smallest error, the smallest normal difference, or the highest distance density, perform a culling operation, update the facet set, and complete the density adjustment.

[0073] Optionally, the S8 specifically includes:

[0074] S81, after completing the update of the facet set of the current image frame and the update of the inter-frame registration graph structure, constructing the output data structure of the current frame;

[0075] S82, for each image frame, extracting the registered facet set and the corresponding inter-frame registration graph structure, and uniformly encoding them into a three-dimensional structure frame;

[0076] S83. Assume that the structure of each frame is represented as , where t represents the time index of the image frame in the video sequence, defining the 3D reconstruction model sequence , where M represents the complete 3D reconstruction model sequence, It represents the structural frame corresponding to the image frame with time index t, and T represents the total number of frames;

[0077] S84, each structural frame The included facet set and inter-frame registration graph structure are stored as a structured data file, marked with time index and recorded as sequence elements;

[0078] S85. Output the three-dimensional reconstruction model sequence M in the order of time index, forming a continuous three-dimensional structure expression result of the dynamic target frame by frame.

[0079] The beneficial effects of the present invention are:

[0080] Compared with existing 3D reconstruction methods, this invention establishes a temporal modeling mechanism for dynamic targets by extracting the target region in each image frame and constructing a facet set containing the 3D spatial position, normal direction, radius value, color vector, and active state label. This improves modeling accuracy and representation integrity. By introducing an inter-frame registration graph structure based on facet modeling, a non-rigid spatial transformation function controlled by the graph structure is used to optimize the alignment of facet sets between consecutive image frames. This solves the registration instability problem faced by traditional methods when dealing with dynamic target deformation, occlusion, and structural changes. The spatial transformation function is constructed using weighted interpolation of graph nodes, giving the registration process both local flexibility and global continuity. By constructing an optimization objective function composed of residual and smoothing terms and solving the spatial transformation parameters using gradient descent, the registration process becomes controllable and highly robust. Furthermore, the invention introduces an error-driven facet density adjustment mechanism in each frame, preserving details in high-error regions while reducing redundant facets in low-error regions, thereby improving the spatial efficiency of structural representation. Finally, by outputting the updated face set and inter-frame registration graph structure frame by frame, a reconstruction model sequence of dynamic targets is constructed, realizing time-series continuous 3D modeling for video sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0082] Figure 1 This is an overall flow chart of the method for real-time 3D reconstruction based on dynamic objects in a video proposed by the present invention;

[0083] Figure 2 This is a schematic diagram of the mechanism of the non-rigid spatial transformation function under graph structure control in the method for real-time 3D reconstruction based on dynamic objects in a video proposed by the present invention;

[0084] Figure 3 This is a schematic diagram of the error-driven surface element density adjustment process of the method for real-time 3D reconstruction based on dynamic targets in videos proposed by the present invention. DETAILED DESCRIPTION

[0085] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0086] refer to Figure 1-3 , a method for real-time 3D reconstruction based on dynamic objects in a video, comprising the following steps:

[0087] S1. Collect video sequences and extract each image frame and the corresponding depth image;

[0088] S2. Perform target region extraction and depth image processing on each image frame to generate an initial facet set;

[0089] S3, constructing an inter-frame registration graph structure based on the SurfelWarp facet modeling structure, performing non-rigid registration using the facet sets of the current image frame and the previous image frame, and updating the facet set of the current image frame;

[0090] S4. Performing surface element activation detection based on the depth change rate and boundary features of the current image frame to determine whether to insert a new surface element, and adding it to the surface element set of the current image frame if the conditions are met;

[0091] S5. In the facet set of the current image frame, if the spatial distance and normal angle between any two facets are both less than a set threshold, they are merged into a new facet;

[0092] S6. Count the number of registrations and occlusion status of each facet in the facet set of the current image frame in consecutive image frames, and delete facets that do not meet the survival condition.

[0093] S7, calculating a reconstruction error map based on the registration results of the current image frame and the previous image frame, and adjusting the bin density of each area of ​​the current image frame according to the reconstruction error map;

[0094] S8. Output the facet set and inter-frame registration graph structure of the current image frame, complete the frame-by-frame 3D reconstruction of the dynamic target in the video, and generate a 3D reconstruction model sequence.

[0095] The present invention introduces a frame-by-frame image and depth image extraction mechanism, combined with a facet-level structural modeling method, to construct a complete dynamic target three-dimensional reconstruction process. Compared with traditional three-dimensional reconstruction algorithms that are only applicable to static scenes or rigid targets, the present invention can perform continuous and high-precision modeling of dynamic objects in the video through non-rigid registration, error-driven density adjustment, and registration graph structure update mechanisms. On the basis of maintaining real-time processing capabilities, it effectively solves problems such as target deformation, local occlusion, and structural instability, and achieves usability and engineering implementation capabilities for complex dynamic scenes. It is suitable for deployment in scenarios with high requirements for continuous spatial expression, such as robot perception, augmented reality, and motion capture.

[0096] In this embodiment, the face element includes a three-dimensional spatial position, a normal direction, a radius value, a color vector, and an active state label.

[0097] The present invention clearly defines that the facet structure should include a three-dimensional spatial position, a normal direction, a radius value, a color vector, and an active state label, so that the facet can not only express geometric information, but also have the ability to fully model the appearance and behavior state. By introducing the radius value, the control of the facet coverage area and boundary reasoning can be supported; by maintaining the color vector, image information can be fused for subsequent rendering or texture recovery; by setting the active state label, the life cycle and retention strategy of the facet can be marked, so that the facet set has the ability to be selectively and dynamically updated between consecutive frames. This structure provides a clear data interface for subsequent alignment, merging, culling, density adjustment and other processing, which improves the expression ability, controllability and scalability of the entire system.

[0098] In this embodiment, S1 specifically includes:

[0099] S11, decoding the video sequence and extracting continuous image frames;

[0100] S12, performing camera intrinsic calibration on each image frame, and obtaining a depth image corresponding to the image frame according to the calibration result;

[0101] S13 , performing time synchronization and pixel-level registration on the image frame and the corresponding depth image to form a one-to-one corresponding image frame and depth image combination.

[0102] The present invention designs a multi-step processing flow of decoding, calibration, and synchronous registration in the image and depth data acquisition process, which effectively improves the spatiotemporal consistency between the image and the depth map, and reduces the error accumulation problem caused by sensor delay or distortion. Through camera intrinsic parameter calibration and pixel-level registration, the present invention ensures that the image frame and the depth image have a one-to-one correspondence during the three-dimensional reconstruction process, laying a rigorous foundation for the construction of surface element coordinates and graph structure. When it comes to high-precision structural calculations and dynamic scene modeling, this processing flow can significantly improve the quality of image depth pairs, improve the stability of the initial modeling stage, and provide a reliable initial state for non-rigid registration and error adjustment.

[0103] In this embodiment, S2 specifically includes:

[0104] S21, performing image enhancement processing based on the grayscale histogram on each image frame to improve the contrast of the edge area of ​​the image;

[0105] S22, performing edge detection based on a gradient operator based on the enhanced image frame to extract the boundary contour of the target area;

[0106] S23, constructing a target area mask in the image frame according to the boundary contour extraction result;

[0107] S24, performing noise filtering and hole filling processing on each depth image to generate a structurally continuous depth image;

[0108] S25, mapping the target area mask to the depth image, and extracting a set of depth values ​​corresponding to the target area;

[0109] S26, calculating a three-dimensional coordinate point according to each pixel position and corresponding depth value in the target area;

[0110] S27: construct a set of all three-dimensional coordinate points to generate an initial facet set corresponding to the current image frame.

[0111] The present invention constructs a complete process from image enhancement, edge detection, mask construction, depth map repair to three-dimensional coordinate generation in terms of target area extraction and depth map processing. Through grayscale histogram enhancement and gradient edge detection, the boundary information of dynamic targets is effectively enhanced, and the regional contour can still be accurately extracted in scenes such as occlusion and lighting changes. The mask is mapped to the depth map, and noise filtering and hole filling are performed to enhance the continuity and accuracy of face element generation. Finally, the three-dimensional coordinates are calculated through depth value and pixel mapping, and the initial face element set of the current frame is constructed, providing accurate, complete, and clearly structured modeling input for subsequent processing of the entire system.

[0112] In this embodiment, S3 specifically includes:

[0113] S31, defining the surfel sets of the current image frame and the previous image frame as the current surfel set and the reference surfel set respectively;

[0114] S32, constructing an inter-frame registration graph structure with each surfel in the current surfel set as a node, and constructing an edge set based on the spatial adjacency relationship between the surfels;

[0115] S33, establishing a one-to-one correspondence between the current surfel set and the reference surfel set based on the coordinate mapping relationship and depth information between the image frames;

[0116] S34, let the three-dimensional position of the i-th facet in the current image frame be , the corresponding reference bin position in the previous image frame is , define the objective function of non-rigid registration:

[0117] ;

[0118] Among them, E is the objective function of non-rigid registration, is the three-dimensional coordinate of the i-th surface element in the current image frame, is the three-dimensional coordinate of the jth surface element in the current image frame, is the corresponding reference bin coordinate in the previous image frame, is the spatial transformation function to be optimized, n is the number of facets in the current image frame, is the set of all edges in the registration graph structure, Indicates the two facet indices that are connected. is the smoothing term weight coefficient, Represents the three-dimensional Euclidean norm, which is used to measure the distance between spatial points;

[0119] S35, the spatial transformation function The structure is:

[0120] ;

[0121] in, To act on the surface element The non-rigid spatial transformation result is For the face element The adjacent graph structure node index set, For the surface element The weight coefficient of node k satisfies the requirement that the sum of all weights is 1. is the three-dimensional rotation matrix corresponding to node k, is the three-dimensional translation vector corresponding to node k, in the expression Represents the node k the imposed local rigid transformation;

[0122] S36, performing iterative optimization based on gradient descent on the objective function E to obtain all variable parameters in the spatial transformation function T;

[0123] S37. For each face element in the current image frame, use the solved spatial transformation function T to update the three-dimensional spatial position, and update the normal direction according to the rotation matrix in the transformation function, while keeping the radius value and color vector unchanged. Combine the updated three-dimensional spatial position, normal direction, radius value, color vector and the original active state label to form a new face element attribute, and generate all transformed face elements in sequence to form a face element set after registration.

[0124] In the non-rigid registration stage, the present invention defines the face element sets of the current frame and the reference frame as nodes in a graph structure and constructs spatial adjacent edges between them, thereby realizing inter-frame matching modeling under structural constraints. By defining an objective function containing data terms and smoothing terms and introducing a spatial transformation function, the registration process is modeled as a graph optimization problem, which effectively improves the flexibility and stability of the registration. The spatial transformation function adopts a local rigid interpolation structure and combines the gradient descent algorithm to solve the variable parameters, ensuring that the registration result of each face element has local responsiveness and overall coherence. Finally, the face element attributes are updated through transformation, and a registered set with continuous structure and high alignment accuracy is generated, thereby improving the temporal consistency and deformation adaptability of the model.

[0125] In this embodiment, the S4 specifically includes:

[0126] S41, performing depth gradient calculation on each pixel position in the current image frame to obtain a depth change rate of each pixel position;

[0127] S42, constructing a three-dimensional spatial mask based on the initial facet set of the current image frame, for identifying all spatial locations represented by the facets;

[0128] S43, traversing all pixel positions in the image frame, determining whether each pixel position is in an area not represented by a bin in the three-dimensional space, and marking pixel positions that meet the uncovered condition as candidate new bin positions;

[0129] S44. Calculate the boundary feature density for all candidate newly added bin positions, and perform bin activation judgment based on the judgment conditions:

[0130] ;

[0131] in, Represents the pixel position in the image frame The depth change rate, Indicates pixel position The density of boundary features near represents the threshold value of the depth change rate, represents the threshold value of boundary feature density, x represents the column index of the pixel, and y represents the row index of the pixel;

[0132] S45. For pixel positions that meet the activation conditions, perform a bin initialization operation to generate three-dimensional coordinates, normal direction, radius, and color value, and form a newly added bin;

[0133] S46: Merge all newly added bins into the bin set of the current image frame to form an updated bin set.

[0134] In the facet activation detection, the present invention integrates the dual criteria of depth change rate calculation and boundary feature density analysis to achieve accurate detection and generation of new facets in uncovered areas. In each frame of the image, pixel-level depth gradient calculation and boundary feature analysis are performed to construct a three-dimensional spatial mask and identify unrepresented candidate areas. By setting the activation threshold condition judgment, it is ensured that facet generation is only performed at locations with high structural changes and boundary information density, effectively avoiding facet redundancy and misjudgment. The newly generated facets contain three-dimensional coordinates, normals, radius and color information, and are incorporated into the current frame facet set, improving the integrity of the overall modeling and edge adaptability.

[0135] In this embodiment, the S5 specifically includes:

[0136] S51. In the facet set of the current image frame, traverse all facet pairs and extract the three-dimensional position and normal direction of each facet pair;

[0137] S52. For each facet, respectively calculate the angle between the spatial Euclidean distance between the three-dimensional positions and the normal direction;

[0138] S53. Assume that the position vectors of the two surface elements are and , the normal direction vectors are and , define the panel merging criterion:

[0139] ;

[0140] in, and represents the three-dimensional coordinates of two surface elements, and represents the corresponding unit normal vector, represents the three-dimensional Euclidean distance between two surface elements, represents the dot product of two normal vectors, Indicates the angle in radians to the normal direction. is the preset spatial distance threshold, is the preset normal angle threshold;

[0141] S54. For all facet pairs that meet both the spatial distance and normal angle thresholds, perform a facet merging operation, perform weighted fusion of the geometry and color attributes of the two facets, and generate a new merged facet.

[0142] S55: Replace the original bins with all the merged bins to form an updated bin set of the current image frame.

[0143] By introducing the dual constraint rules of spatial distance and normal angle, the present invention designs an accurate facet merging judgment and attribute fusion strategy, which effectively solves the problems of facet density redundancy and structural duplication. When performing facet pair traversal, the Euclidean distance of the spatial position and the angle of the normal direction are calculated respectively, and the merging judgment is made according to the set threshold. The weighted fusion of geometric and color attributes is performed on the facet pairs that meet the conditions to ensure that the new facets are representative in terms of spatial position and appearance expression. Finally, the original facet set is replaced with the new facet to improve the compression efficiency, computational burden and visual consistency of the facet representation, laying a stable foundation for subsequent dynamic modeling and output.

[0144] In this embodiment, S6 specifically includes:

[0145] S61. Create a registration record list for each facet in the facet set of the current image frame, recording the number of times each facet participates in registration in consecutive image frames and the number of times each facet is marked as occluded in consecutive image frames;

[0146] S62. For each facet, count the total number of registration participations and the total number of occlusion marks;

[0147] S63: Setting a minimum allowable number of registration times and a maximum allowable number of occlusion times. When the number of registration times of a facet is less than the minimum allowable value, or the number of occlusion times of a facet is greater than the maximum allowable value, the facet is determined to not meet the retention requirement.

[0148] S64, deleting the facets that do not meet the retention requirement from the facet set of the current image frame;

[0149] S65: Retain the bins that meet the retention requirement and form an updated bin set of the current image frame.

[0150] This invention introduces a joint evaluation strategy for the number of registrations and occlusions in the facet rejection and retention mechanism. By constructing a multi-frame record list and statistically analyzing historical interaction information, the facet's value is determined. By setting two thresholds, the minimum number of registrations and the maximum number of occlusions, a comprehensive determination is made as to whether a facet meets the retention criteria. Eliminating facets that do not meet the criteria effectively reduces redundant data interference and improves the compactness and structural consistency of the set. At the same time, retaining high-confidence facets with stable structures and minimal occlusions enhances structural continuity between consecutive frames, making the reconstructed model both temporally robust and capable of long-term stable expression.

[0151] In this embodiment, the S7 specifically includes:

[0152] S71. Calculate a 3D registration error value for each facet in the facet set of the current image frame, and construct a reconstruction error map.

[0153] S72, dividing the image frame into a set of spatial grid regions, and counting the number of facets and the average error value for each spatial grid region;

[0154] S73. Let the average error value of each spatial grid area be , the current number of facets is d, and the error threshold is , the lower limit of the surface element density is , the upper limit of the surface element density is , define the surface element density adjustment function:

[0155] ;

[0156] in, represents the number of facets in the spatial grid area that should be adjusted, is the increase factor when the error is too high, is the reduction factor when the error is too low;

[0157] S74. For the spatial grid area that meets the addition conditions, interpolation sampling is performed based on the depth map and boundary features in the current image frame to generate the three-dimensional position and normal direction of the newly added surface element;

[0158] S75. For the spatial grid area that meets the reduction condition, select the facets with the smallest error, the smallest normal difference, or the highest distance density, perform a culling operation, update the facet set, and complete the density adjustment.

[0159] By constructing a reconstruction error map and a spatial grid division mechanism, the present invention establishes a dynamic facet density adjustment strategy based on regional error statistics, which enables the modeling system to have adaptive resource scheduling and structural optimization capabilities. In each frame, the registration errors of all facets are calculated, and the image is divided into spatial regions. The number of facets and the average error are counted for each region. Based on the error adjustment function, it is determined whether the region needs to be densified or de-densified. Facet interpolation and addition are performed in areas with high errors, and redundant elimination is performed in areas with low errors. This strategy ensures a balanced distribution of the overall reconstruction quality and improves the system modeling efficiency and the ability to control local expression details.

[0160] In this embodiment, S8 specifically includes:

[0161] S81, after completing the update of the facet set of the current image frame and the update of the inter-frame registration graph structure, constructing the output data structure of the current frame;

[0162] S82, for each image frame, extracting the registered facet set and the corresponding inter-frame registration graph structure, and uniformly encoding them into a three-dimensional structure frame;

[0163] S83. Assume that the structure of each frame is represented as , where t represents the time index of the image frame in the video sequence, defining the 3D reconstruction model sequence , where M represents the complete 3D reconstruction model sequence, Indicates the structure frame corresponding to the image frame with time index t, and T represents the total number of frames;

[0164] S84, each structural frame The included facet set and inter-frame registration graph structure are stored as a structured data file, marked with time index and recorded as sequence elements;

[0165] S85. Output the three-dimensional reconstruction model sequence M in the order of time index, forming a continuous three-dimensional structure expression result of the dynamic target frame by frame.

[0166] The present invention constructs a unified encoding mechanism for facet sets and registration graph structures in the output stage, forming a complete 3D reconstruction model sequence through structural frame construction and time index organization. The structural information of each frame is packaged into a standard structural frame, containing the facet attributes and graph structure connection relationship, and is output as a continuous sequence in time index order. This mechanism not only enables the storage and reuse of frame-by-frame reconstruction results, but also improves the efficiency and accuracy of subsequent structural analysis, 3D visualization, and cross-frame alignment tasks. It is suitable for practical application scenarios with high temporal continuity requirements, such as motion capture, dynamic modeling, and time-consistent reconstruction systems.

[0167] Example 1:

[0168] To verify the feasibility of the present invention in implementation, the present invention is applied to the dynamic human 3D modeling scene in a smart security system. Reconstruction tests and performance analysis are carried out in an actual complex environment to comprehensively evaluate the performance of the proposed method in terms of structural integrity, registration accuracy, real-time performance, and stability.

[0169] The experiment took place in a public open area on the first floor of a large commercial complex in Nanjing, Jiangsu Province. This densely populated area, with frequently changing backgrounds, presents typical challenges for dynamic object reconstruction. The system deployed an Intel RealSense D455 depth camera, mounted 2.5 meters above the ground with a 30-degree pitch angle to cover the approximately 8-by-6-meter atrium. The test was scheduled for March 15, 2025, from 10:00 AM to 11:00 AM. The scene included a variety of human behavior patterns, including natural walking, cross-occlusion, and turns and pauses.

[0170] In the proposed system, an RGB-D video stream is first synchronously captured at 30 frames per second and a resolution of 1280×720. The system automatically decodes the image frames, calibrates the camera, and performs pixel-level alignment. The system then performs grayscale histogram enhancement, edge extraction, and object mask construction on the image. Simultaneously, hole filling and edge restoration operations are performed on the depth map to jointly generate an initial set of surface elements for the current frame. Each surface element contains five structural attributes: 3D spatial position, normal direction, radius value, color vector, and active state label.

[0171] During the inter-frame processing phase, the system establishes a registration graph based on the SurfelWarp structure and performs non-rigid registration using a local rigid interpolation mechanism between the nodes of the graph structure. For each frame and the facet set of the previous frame, an optimization objective function is constructed. The rotation and translation parameters in the spatial transformation function are solved by gradient descent, and the position and normal direction of each facet in the facet set are updated accordingly. The system further uses the depth change rate and boundary density to determine uncovered areas, inserting new facests. The system also calculates the distribution of the error map and removes facests in highly redundant areas, maintaining a balance between structural accuracy and modeling efficiency.

[0172] To verify the performance of the proposed method in dynamic scenes, we introduced two mainstream comparison methods: Method A, a voxel-fusion-based TSDF reconstruction, and Method B, an ICP-based point cloud stitching algorithm, for horizontal testing against the proposed method. Each method was run on the same video dataset, and the evaluation metrics included structural integrity score, average reconstruction error, frame rate, redundancy rate, and data processing delay. The results are shown in Table 1:

[0173] Table 1 Comparison of structure reconstruction performance of different 3D reconstruction methods in dynamic target scenes

[0174] ;

[0175] Results show that the proposed method significantly outperforms traditional methods across all metrics, particularly in structural integrity and error control. The structural score improved by over 25%, with the average error kept within 5.2 mm. While maintaining a real-time runtime of nearly 20 frames per second, the method effectively reduces the number of redundant facets generated, improving overall modeling efficiency.

[0176] In addition, we selected some key frame indexes and recorded the structural frame performance of the system output, including the number of surfels, the number of activations and cullings, and the registration error, to verify the temporal stability and continuity of its output structure, as shown in Table 2:

[0177] Table 2 Continuity and stability tracking data of the output structure of the 3D reconstruction model sequence

[0178] ;

[0179] As shown in Table 2, the system exhibited no glitches or registration failures in consecutive keyframes during testing, and the structured output frames maintained high density, consistency, and integrity. The number of activated and eliminated surfels dynamically adjusted between frames, maintaining a stable reconstruction error within a range of 5–6 mm with minimal fluctuation between frames. This demonstrates that the system can effectively handle challenges posed by occlusion, motion blur, and object deformation, demonstrating its high stability and engineering controllability.

[0180] In summary, this embodiment verifies that the present invention has a high degree of structural expression ability, real-time performance and timing consistency in a real dynamic environment. It can solve the problems of structural fracture, registration drift, and facet redundancy in the existing technology in dynamic video scenes. It has broad practical prospects and can be promoted and applied to various real-time three-dimensional modeling tasks such as smart security, human body capture, augmented reality, and robot navigation.

[0181] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for real-time 3D reconstruction based on dynamic objects in a video, characterized in that: The steps include: S1. Collect video sequences and extract each image frame and the corresponding depth image; S2. Perform target region extraction and depth image processing on each image frame to generate an initial facet set; S3, constructing an inter-frame registration graph structure based on the SurfelWarp facet modeling structure, performing non-rigid registration using the facet sets of the current image frame and the previous image frame, and updating the facet set of the current image frame; S4. Performing surface element activation detection based on the depth change rate and boundary features of the current image frame to determine whether to insert a new surface element, and adding it to the surface element set of the current image frame if the conditions are met; S5. In the facet set of the current image frame, if the spatial distance and normal angle between any two facets are both less than a set threshold, they are merged into a new facet; S6. Count the number of registrations and occlusion status of each facet in the facet set of the current image frame in consecutive image frames, and delete facets that do not meet the survival condition. S7, calculating a reconstruction error map based on the registration results of the current image frame and the previous image frame, and adjusting the bin density of each area of ​​the current image frame according to the reconstruction error map; S8, outputting the facet set and inter-frame registration graph structure of the current image frame, completing frame-by-frame 3D reconstruction of the dynamic target in the video, and generating a 3D reconstruction model sequence; The S4 specifically includes: S41, performing depth gradient calculation on each pixel position in the current image frame to obtain a depth change rate of each pixel position; S42, constructing a three-dimensional spatial mask based on the initial facet set of the current image frame, for identifying all spatial locations represented by the facets; S43, traversing all pixel positions in the image frame, determining whether each pixel position is in an area not represented by a bin in the three-dimensional space, and marking pixel positions that meet the uncovered condition as candidate new bin positions; S44. Calculate the boundary feature density for all candidate newly added bin positions, and perform bin activation judgment based on the judgment conditions: ； in, Represents the pixel position in the image frame The depth change rate, Indicates pixel position The density of boundary features near represents the threshold value of the depth change rate, represents the threshold of boundary feature density, represents the column index of the pixel, Represents the row index of the pixel; S45. For pixel positions that meet the activation conditions, perform a bin initialization operation to generate three-dimensional coordinates, normal direction, radius, and color value, and form a newly added bin; S46, merging all newly added bins into the bin set of the current image frame to form an updated bin set; The S7 specifically includes: S71. Calculate a 3D registration error value for each facet in the facet set of the current image frame, and construct a reconstruction error map. S72, dividing the image frame into a set of spatial grid regions, and counting the number of facets and the average error value for each spatial grid region; S73. Let the average error value of each spatial grid area be , the current number of bins is , set the error threshold to , the lower limit of the surface element density is , the upper limit of the surface element density is , define the surface element density adjustment function: ； in, represents the number of facets in the spatial grid area that should be adjusted, is the increase factor when the error is too high, is the reduction factor when the error is too low; S74. For the spatial grid area that meets the addition conditions, interpolation sampling is performed based on the depth map and boundary features in the current image frame to generate the three-dimensional position and normal direction of the newly added surface element; S75. For the spatial grid area that meets the reduction condition, select the facets with the smallest error, the smallest normal difference, or the highest distance density, perform a culling operation, update the facet set, and complete the density adjustment.

2. The method for real-time 3D reconstruction based on dynamic objects in a video according to claim 1, characterized in that: The face element includes a three-dimensional space position, a normal direction, a radius value, a color vector, and an active state label.

3. The method for real-time 3D reconstruction based on dynamic objects in a video according to claim 1, characterized in that: Said S1 specifically includes: S11, decoding the video sequence and extracting continuous image frames; S12, performing camera intrinsic calibration on each image frame, and obtaining a depth image corresponding to the image frame according to the calibration result; S13 , performing time synchronization and pixel-level registration on the image frame and the corresponding depth image to form a one-to-one corresponding image frame and depth image combination.

4. The method for real-time 3D reconstruction based on dynamic objects in a video according to claim 1, characterized in that: The S2 specifically includes: S21, performing image enhancement processing based on the grayscale histogram on each image frame to improve the contrast of the edge area of ​​the image; S22, performing edge detection based on a gradient operator based on the enhanced image frame to extract the boundary contour of the target area; S23, constructing a target area mask in the image frame according to the boundary contour extraction result; S24, performing noise filtering and hole filling processing on each depth image to generate a structurally continuous depth image; S25, mapping the target area mask to the depth image, and extracting a set of depth values ​​corresponding to the target area; S26, calculating a three-dimensional coordinate point according to each pixel position and corresponding depth value in the target area; S27: construct a set of all three-dimensional coordinate points to generate an initial facet set corresponding to the current image frame.

5. The method for real-time 3D reconstruction based on dynamic objects in a video according to claim 1, characterized in that: The S3 specifically includes: S31, defining the surfel sets of the current image frame and the previous image frame as the current surfel set and the reference surfel set respectively; S32, constructing an inter-frame registration graph structure with each surfel in the current surfel set as a node, and constructing an edge set based on the spatial adjacency relationship between the surfels; S33, establishing a one-to-one correspondence between the current surfel set and the reference surfel set based on the coordinate mapping relationship and depth information between the image frames; S34, set the current image frame The three-dimensional position of the surface element is , the corresponding reference bin position in the previous image frame is , define the objective function of non-rigid registration: ； in, is the objective function of non-rigid registration, The current image frame The three-dimensional coordinates of the surface elements, The current image frame The three-dimensional coordinates of the surface elements, is the corresponding reference bin coordinate in the previous image frame, is the spatial transformation function to be optimized, is the number of bins in the current image frame, is the set of all edges in the registration graph structure, Indicates the two facet indices that are connected. is the smoothing term weight coefficient, Represents the three-dimensional Euclidean norm, which is used to measure the distance between spatial points; S35, the spatial transformation function The structure is: ； in, To act on the surface element The non-rigid spatial transformation result is For the face element The adjacent graph structure node index set, For the surface element For Node The weight coefficient satisfies that the sum of all weights is 1. For nodes The corresponding three-dimensional rotation matrix, For nodes The corresponding three-dimensional translation vector; S36, for the objective function Perform iterative optimization based on gradient descent to solve the spatial transformation function All variable parameters in; S37. For each face element in the current image frame, use the solved spatial transformation function T to update the three-dimensional spatial position, and update the normal direction according to the rotation matrix in the transformation function, while keeping the radius value and color vector unchanged. The updated three-dimensional spatial position, normal direction, radius value, color vector and the original active state label are combined to form a new face element attribute, and all transformed face elements are generated in sequence to form a registered face element set.

6. The method for real-time 3D reconstruction based on dynamic objects in a video according to claim 1, characterized in that: The S5 specifically includes: S51. In the facet set of the current image frame, traverse all facet pairs and extract the three-dimensional position and normal direction of each facet pair; S52. For each facet, respectively calculate the angle between the spatial Euclidean distance between the three-dimensional positions and the normal direction; S53. Assume that the position vectors of the two surface elements are and , the normal direction vectors are and , define the panel merging criterion: ； in, and represents the three-dimensional coordinates of two surface elements, and represents the corresponding unit normal vector, represents the three-dimensional Euclidean distance between two surface elements, represents the dot product of two normal vectors, Indicates the angle in radians to the normal direction. is the preset spatial distance threshold, is the preset normal angle threshold; S54. For all facet pairs that meet both the spatial distance and normal angle thresholds, perform a facet merging operation, perform weighted fusion of the geometry and color attributes of the two facets, and generate a new merged facet. S55: Replace the original bins with all the merged bins to form an updated bin set of the current image frame.

7. The method for real-time 3D reconstruction based on dynamic objects in a video according to claim 1, characterized in that: The S6 specifically includes: S61. Create a registration record list for each facet in the facet set of the current image frame, recording the number of times each facet participates in registration in consecutive image frames and the number of times each facet is marked as occluded in consecutive image frames; S62. For each facet, count the total number of registration participations and the total number of occlusion marks; S63: Setting a minimum allowable number of registration times and a maximum allowable number of occlusion times. When the number of registration times of a facet is less than the minimum allowable value, or the number of occlusion times of a facet is greater than the maximum allowable value, the facet is determined to not meet the retention requirement. S64, deleting the facets that do not meet the retention requirement from the facet set of the current image frame; S65: Retain the bins that meet the retention requirement and form an updated bin set of the current image frame.

8. The method for real-time 3D reconstruction based on dynamic objects in a video according to claim 1, characterized in that: The S8 specifically includes: S81, after completing the update of the facet set of the current image frame and the update of the inter-frame registration graph structure, constructing the output data structure of the current frame; S82, for each image frame, extracting the registered facet set and the corresponding inter-frame registration graph structure, and uniformly encoding them into a three-dimensional structure frame; S83. Assume that the structure of each frame is represented as ,in Represents the time index of the image frame in the video sequence and defines the 3D reconstruction model sequence ,in, represents a complete 3D reconstruction model sequence, Indicates that the time index is The structure frame corresponding to the image frame, Indicates the total number of frames; S84, each structural frame The included facet set and inter-frame registration graph structure are stored as a structured data file, marked with time index and recorded as sequence elements; S85: Output the 3D reconstruction model sequence in time index order , forming a continuous three-dimensional structure expression result for dynamic targets frame by frame.

Images