Dynamic 3D model reconstruction method and system based on high-frequency implicit network and skin model

Abstract

The invention discloses a dynamic 3D model reconstruction method and system based on a high-frequency implicit network and a skin model. The method comprises the steps that multiple segments of original videos are input and preprocessed; constructing a standard shape model by using a high-frequency implicit network, and extracting density, standard embedding and color features from each preprocessed video frame; inputting the extracted density, standard embedding and color features mapped into the standard shape model into a neural hybrid skin deformation model, and performing pose modeling and deformation control; establishing a corresponding relation between pixel points and a unified standard three-dimensional space in multi-frame image data, and optimizing model parameters through standard matching; and performing volume rendering on the output characteristics of the high-frequency implicit network and the neural hybrid skin deformation model, and outputting a reconstructed dynamic three-dimensional model.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and computer vision, and in particular to a dynamic 3D model reconstruction method, specifically a dynamic 3D model reconstruction method and system based on a high-frequency implicit network and a skinned model. Background Technology

[0002] 3D reconstruction technology has always been a hot topic in computer graphics and computer vision. Traditional 3D reconstruction techniques are mainly based on multi-view geometry principles, recovering 3D structural information by analyzing images from different perspectives. Structured light methods, Time-of-Flight (TOF) technology, and triangulation were commonly used early 3D reconstruction methods. These techniques usually require specialized hardware, such as structured light projectors or depth cameras, to obtain depth information by measuring the reflection time or shift of light, but they are easily limited by equipment.

[0003] With the development of technology, reconstructing high-fidelity 3D models from multi-view images can achieve more realistic and smooth visual effects, demonstrating great application prospects in fields such as medical health, robotics, AR and VR, and cultural heritage protection. The MVS (Multi-View Stereo Vision) method reconstructs 3D models from a series of RGB images at different angles. Multi-view information can generate dense point clouds, suitable for complex surface reconstruction. However, traditional MVS assumes a static scene, and dynamic objects can lead to matching errors, requiring combination with motion segmentation or temporal modeling.

[0004] The development of deep learning has continuously contributed to 3D reconstruction. In 2020, Ben Mildenhall et al. proposed the NeRF model, which uses a fully connected neural network to model the density and color of a scene, and then synthesizes a high-quality 2D view using ray tracing technology. Since then, researchers have made many extensions and improvements to address its limitations in practical applications, and significant progress has been made in dynamic modeling. In recent years, neural implicit surface representation has received widespread attention due to its resolution independence, continuity, and differentiability. By implicitly defining the SDF (signed distance field) using a multilayer perceptron, the resource consumption of explicitly storing voxel grids can be avoided, and high-precision queries at arbitrary locations and efficient calculation of analytical gradients can be achieved. This method shows significant advantages in handling complex shapes and deformations, becoming an important research direction in the field of 3D reconstruction. In handling the dynamic modeling of articulated objects, linear hybrid skinning technology is widely used. By using the skinning model to control the skeleton, the motion and deformation of dynamic objects can be effectively simulated, thereby improving the stability and accuracy of reconstruction.

[0005] However, current methods still have limitations in handling complex deformations and topological changes, and it is difficult to effectively capture detailed information in dynamic scenes. Therefore, research in the field of 3D reconstruction is still of great significance. Summary of the Invention

[0006] To address the 3D reconstruction problem in existing technologies, this invention proposes a dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models. By combining high-frequency implicit networks and skinning models, the accuracy and stability of deformation expression are improved. This method can be applied to capture detailed features in videos and perform highly realistic dynamic 3D reconstruction.

[0007] According to one aspect of the present invention, a method for dynamic 3D model reconstruction based on high-frequency implicit networks and skinning models is provided, comprising: Input multiple raw video segments and perform preprocessing; A canonical shape model is constructed using a high-frequency implicit network, and density, canonical embedding, and color features are extracted from each preprocessed video frame. The extracted density, canonical embedding, and color features, which have been mapped to the canonical shape model, are input into the neural hybrid skin deformation model for pose modeling and deformation control. Establish the correspondence between pixels and a unified standard 3D space in multi-frame image data, and optimize model parameters through standard matching; Volumetric rendering is performed on the output features of the high-frequency implicit network and the neural hybrid skin deformation model to output the reconstructed dynamic 3D model.

[0008] As a further technical solution, the preprocessing includes: Camera parameters are initialized, and the rendering range for each frame is dynamically adjusted according to the projection depth of the standard spatial surface. Root pose initialization: Train a CNN model, input the DensePose CSE feature map, predict the root pose for each frame, and align the target to the normal space. Pixel feature embedding uses the DensePose CSE model, a deep learning-based human pose estimation model, to extract pixel-level feature embeddings, providing semantic cues for dense correspondences across videos.

[0009] As a further technical solution, density, normalized embedding, and color features are extracted from each preprocessed video frame, including: Two concatenated quadratic convolutional layers are used to replace the linear layers in the SIREN network, and second-order constraint terms are introduced. An optimized SIREN network is used to perform density, canonical embedding, and color feature extraction on each video frame.

[0010] As a further technical solution, pose modeling and deformation control are performed, including: Determining arbitrary 3D points using the skeleton transformation module Its corresponding 3D point Mapping relationships in the normative space; The root pose and body pose are modeled using a pose representation module. Use the skin weight module to generate skin weights for 3D skeletal animation.

[0011] As a further technical solution, the skin weighting module also includes: The roughness weight module calculates roughness skin weights by explicitly modeling the local influence region of the skeleton using a 3D Gaussian ellipsoid. The detail correction module uses an optimized SIREN network to capture high-frequency, complex geometric details that are difficult to capture with Gaussian weights.

[0012] As a further technical solution, a correspondence between pixels and a standardized three-dimensional space is established in multi-frame image data, including: The pixel positions and features in each frame are mapped to a unified three-dimensional normed space and matched with the embeddings of other frames to achieve consistent alignment across viewpoints and time.

[0013] As a further technical solution, volumetric rendering is performed on the output features of the high-frequency implicit network and the neural hybrid skin deformation model, including: Given a pixel in one frame of an image, predict the color or position change of that pixel in another frame and reconstruct its position in 3D space.

[0014] According to one aspect of the present invention, a dynamic 3D model reconstruction system based on a high-frequency implicit network and a skinning model is provided, comprising: The first main module is used to input multiple raw video segments and perform preprocessing. The second main module is used to construct a canonical shape model using a high-frequency implicit network and extract density, canonical embedding, and color features from each preprocessed video frame. The third main module is used to input the extracted density, normalized embedding and color features that have been mapped to the normalized shape model into the neural hybrid skin deformation model for pose modeling and deformation control. The fourth main module is used to establish the correspondence between pixels and a unified standard three-dimensional space in multi-frame image data, and to optimize model parameters through standard matching. The fifth main module is used to perform volume rendering on the output features of the high-frequency implicit network and the neural hybrid skin deformation model, and output the reconstructed dynamic 3D model.

[0015] According to one aspect of the present invention, a dynamic 3D model reconstruction device based on a high-frequency implicit network and a skinned model is provided, comprising a memory and a processor, wherein the memory stores program instructions that are executed by the processor, and the processor invokes the program instructions to execute the dynamic 3D model reconstruction method based on a high-frequency implicit network and a skinned model.

[0016] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the dynamic 3D model reconstruction method based on a high-frequency implicit network and skinning model.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention first inputs multiple original video segments and preprocesses them; then, it utilizes an optimized high-frequency implicit network to perform geometric detail modeling on the input data, constructing a standardized shape model and extracting density, standardized embedding, and color features; next, it uses a neural hybrid skin deformation model to handle large-scale deformation and pose changes; subsequently, it achieves cross-view spatiotemporal consistency alignment using a standardized matching module; finally, it outputs a high-fidelity dynamic 3D model through a volumetric rendering module. The neural hybrid skin deformation model provided by this invention includes a skeleton transformation module, a pose representation module, and a skin weighting module, combining an explicit Gaussian model with an implicit correction strategy to improve the accuracy and stability of deformation representation. The method described in this invention can effectively capture detailed changes in dynamic scenes and has significant application value in fields such as video-based 3D reconstruction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models provided in an embodiment of the present invention. Detailed Implementation

[0020] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0022] This invention provides a dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models, such as... Figure 1 As shown, it includes: Step S1: Input multiple raw video segments and perform preprocessing; Step S2: Construct a canonical shape model using a high-frequency implicit network, and extract density, canonical embedding, and color features from each preprocessed video frame; Step S3 involves inputting the extracted density, normalized embedding, and color features mapped to the normalized shape model into the neural hybrid skin deformation model for pose modeling and deformation control. This step uses the neural hybrid skin deformation model to handle a wide range of deformations and movements. The neural hybrid skin deformation model includes a skeleton transformation module, a pose representation module, and a skin weight module. The skin weight module includes a coarsening weight module and a detail correction module. Step S4: Establish the correspondence between pixels and a unified standard three-dimensional space in multi-frame image data, and optimize model parameters through standard matching; Step S5: Perform volume rendering on the output features of the high-frequency implicit network and the neural hybrid skin deformation model to output the reconstructed dynamic 3D model.

[0023] Preferably, in step S1, the input raw video is preprocessed, including camera parameter initialization, root pose initialization, and pixel feature embedding. Camera parameter initialization dynamically adjusts the rendering range of each frame based on the projection depth of the canonical space surface to avoid invalid sampling. Root pose initialization involves performing DensePose CSE inference on the raw video frames to obtain DensePose CSE feature maps, which are used for root pose initialization and pixel feature embedding. A CNN model is trained, inputting the DensePose CSE feature maps to predict the root pose of each frame and align the target to the canonical space. This method aligns the target in the video frame with the canonical space, providing an initial alignment reference for subsequent 3D reconstruction. Pixel feature embedding uses the DensePose CSE model to extract pixel-level feature embeddings, providing semantic cues for dense correspondences across videos.

[0024] Preferably, in step S2, the canonical shape model is defined as a time-invariant canonical space, serving as a unified reference space to accommodate the alignment and fusion of density, canonical embedding, and color features in subsequent frames, ensuring consistency across viewpoints and time periods. Therefore, the canonical shape model is not only constructed in step S2 but also provides a reference coordinate system during the neural hybrid skinning and volumetric rendering processes in steps S3 to S5.

[0025] Preferably, in step S2, the high-frequency implicit network, based on the SIREN network, replaces its linear layers with cascaded double-layer quadratic convolutional layers and introduces second-order constraint terms to stabilize the optimization, thereby forming an optimized SIREN structure that can capture high-frequency geometric details. Its overall structure is an improved version of SIREN, and the specific formula is as follows: , Where x is the input vector, representing the spatial coordinates or feature vector received by the network. , , For learnable parameter weight matrix, , , For learnable bias parameters, Represents the element-wise product (Hadamard product).

[0026] A second-order constraint term is introduced to stabilize the optimization process. The specific calculation formula is as follows: , in The first gradient of the input function. The input function is the Hessian matrix (second derivative). The space containing the object to be rendered (the domain of the integral) is represented by T, and T represents the transpose.

[0027] The optimized SIREN function is used to extract density, canonical embedding, and color features for each video frame. Canonical embedding serves as a cross-frame matching feature, representing the position or encoding of points within the video frame in the canonical shape model. Density and color are physical properties in canonical spatial coordinates. These three features are used together as input features for subsequent skinning deformation, matching, and rendering. The specific formula is as follows: , Where c represents color, σ is density, and ψ is normalized embedding. To change the direction of the view, Represents a learnable environment code. Represents the coordinates of a three-dimensional point in normal space (transforming normal coordinate points into normal coordinate points through mapping); This represents a differentiable transformation that maps the SDF output to the volume density (a non-negative scalar), with parameter λ (this parameter is usually learnable), and input is... The output is a non-negative real number 𝜎, which can be directly used as the density in the volume rendering / volume absorbing model to calculate the opacity 𝛼 or rendering weight.

[0028] Preferably, in step S3, the density, canonical embedding, and color features extracted in step S2 through a high-frequency implicit network are used as input to the neural hybrid skin deformation model for pose modeling and deformation control. Specifically, this involves finding the shape through skeletal transformation. and The mapping between them involves a skeleton, a virtual rigid structure connected hierarchically to form a skeletal system. Each bone corresponds to a joint, and the movement of a bone will affect the connected joints. At a given time t, for any 3D point... Its corresponding 3D point They have a certain mapping relationship in the gauge space, and the specific formula is as follows: .

[0029] Similarly, Transform back to points in normal space The specific formula is as follows: , The global rigid body transformation operator for the skeleton at time t is a matrix representation that combines the current rigid body poses of each bone calculated at the bone level to transform the points in gauge space to the coordinate system at time t (or from time t to gauge space). It is a learnable parameter.

[0030] Preferably, in step S3, the skeleton transformation module, and These are the rigid body transformation matrices that transform the skeleton from its resting configuration to its current configuration, and their corresponding inverse transformations. The transformations are obtained by a weighted average of B rigid body transformations, as shown in the following formula: , in, and It is the bone weight associated with bone b, which will be obtained from the skin weight module. Let represent the rigid body transformation matrix of the b-th bone at time t.

[0031] Preferably, in step S3, the pose representation module includes two aspects: root pose and body pose. The pose changes of dynamic objects contain rapidly changing high-frequency components. Traditional discrete pose representation or linear interpolation is difficult to accurately model continuous and complex high-frequency motion. Therefore, for a given time t, the pose is modeled through an optimized SIREN network in a high-frequency implicit network to further enhance the detail capture capability. The modeled pose is the driving force for the skeleton-skin transformation, mainly realizing the transformation and inverse transformation from the normal space to the observation space in the neural hybrid skin deformation, that is, playing a role in the mapping link from the normal space to the current frame space, and being continuously corrected in the final differentiable rendering and optimization. The specific formula is: , in and These are the latent codes for the root pose and body pose at time t, respectively. Each dimension of this latent code is represented as a linear combination of sinusoidal basis functions, as shown in the following formula: , in It is a one-dimensional basis of sine and cosine, whose frequency increases linearly on a logarithmic scale. Learn a separate meta-weight matrix for each video ∈{1...,M}.

[0032] Specifically, this network represents root pose and body pose through latent codes, where the dimension of each latent code is composed of a linear combination of sinusoidal basis functions. These sinusoidal basis functions increase in frequency linearly on a logarithmic scale, and a unique meta-weight matrix is ​​learned for each video. This method utilizes a combination of sine and cosine functions to efficiently encode pose information and accurately capture dynamic changes.

[0033] Preferably, in step S3, the skinning weight module mainly involves generating suitable skinning weights for 3D skeletal animation, combining an explicit Gaussian ellipsoid and an implicit correction term, and introducing a mixing coefficient. ∈[0,1], where It can be dynamically adjusted during the training phase. In the early stages of training, explicit Gaussian weights are used. The primary function is to provide a stable initial weight distribution, preventing the implicit network from getting trapped in local optima due to random initialization. As the training process progresses, implicit correction terms are gradually added. The weights are calculated using the following formula: , in, Indicates softmax ( ).

[0034] Preferably, in step S3, the roughness weight module calculates the roughness skinning weight by explicitly modeling the local influence region of the skeleton using a 3D Gaussian ellipsoid. That is, the Gaussian skinning weight is determined by the Mahalanobis distance between X and the Gaussian ellipsoid; the smaller the distance, the greater the influence of bone b on point X. The specific formula is as follows: , in The center of the skeleton It is a precision matrix, which is composed of the orientation of the bones. The Gaussian component will dynamically adjust the precision matrix according to the deformation of the object, thereby optimizing the detail restoration during the bone transformation process. The orientation matrix of the ellipsoid corresponding to bone b is a 3x3 rotation matrix; This represents the precision diagonal matrix along these principal axes, i.e., the inverse variance in each principal axis direction. ); yes The transpose of is used for reverse rotation.

[0035] Preferably, in step S3, the detail correction module uses coordinated and optimized SIREN learning to capture complex geometric details that are difficult to capture by Gaussian weights. The specific formula is as follows: , in , These are the coordinates and pose codes of a 3D point, respectively.

[0036] Preferably, in step S4, the canonical matching module is used to establish a correspondence between pixels and a unified canonical 3D space in multi-frame image data. By mapping the pixel position and its features in any frame of image to a unified 3D canonical space and matching it with the embedding features of corresponding pixels in other frames, consistent alignment across time and viewpoints is achieved. Combined with the backpropagation of the matching loss function, the implicit function parameters, embedding representation parameters, and skeleton skinning weight parameters involved in the network can be updated and optimized, thereby ensuring the accuracy of the correspondence across frames and viewpoints. Given two-dimensional pixels in an image frame... The goal is to find the corresponding position in the canonical space. This is achieved by performing a weighted average of the matching degree between the 2D pixel and all 3D points, yielding the most likely 3D point location. The specific formula is: , in, For 2D pixel coordinates in an image frame, To normalize a candidate 3D point in space, A set of 3D points uniformly sampled in a normal space. For each 3D sampling point It indicates that the point is related to a 2D pixel. The normalized score of the match.

[0037] Preferably, in step S5, the volume rendering module uses the canonical spatial features jointly generated by the high-frequency implicit network and the neural hybrid skinning module as input for volume rendering calculation. Specifically, the input includes color features, density features, and embedding features output by the high-frequency implicit network, and pose parameters and skinning weights output by the neural hybrid skinning module. Given a pixel in a frame of an image... It can predict the color or position change of a pixel in another frame and reconstruct its position in 3D space. It is along the ray from The i-th sample point is emitted. Each ray deforms along this point, and here a neural hybrid skin deformation model is used to "pull it back" to the normal space to obtain the normal space coordinates. In the normal space, each point All inputs are fed into the optimized SIREN network, where multiple sampling is used to calculate the penetration of light through the scene. The specific formula is as follows: , in: It is the flight probability of the i-th sample point, that is, the probability of a photon being transmitted between the camera and the sample point. This represents the probability that the i-th segment is not absorbed, that is, the probability that the photon passes through the segment and continues to move forward; Similarly, represents the j-th segment. Let represent the volume density at the i-th sampling point, and let represent the rate of photon absorption per unit length. is the length of the i-th segment, representing the spatial distance along the ray on that sampling segment.

[0038] Based on the sampling point density obtained in the high-frequency implicit network, the photon density at that location can be calculated. The transmission probability of the interval, and the probability of photons propagating between the camera and the sampling point. This step transforms spatial density into weights, preparing for weighted calculations of color, position, etc., thereby obtaining the color and transparency of each sampling point. The specific formula is: , Where N is the number of sampling points.

[0039] Then, using the weights and canonical space coordinates of each sampling point, the expected surface intersection position is calculated, yielding the expected corresponding surface point of the current pixel in the canonical space. The specific formula is as follows: , Then take each sampling point from the previous step Transformed to time t′, its pixel position in the new image is obtained by projecting it through the camera and weighting it, using the following formula: , Further calculations yield the final volumetric rendering formula from the 2D optical flow representing time t to t′. The specific formula is as follows: .

[0040] The volumetric rendering module is used to predict the color or position change of a pixel in an image in another frame and reconstruct its position in 3D space. Specifically, multiple sample points are emitted from a given pixel along a ray, and these points are "pulled back" to the canonical space using a neural hybrid skinning model. In the canonical space, each sample point is input into an optimized SIREN network, and the penetration of light in the scene is calculated through multiple samplings. This process involves calculating the flight probability of each sample point to obtain the transmission probability of photons between two time points, and converting spatial density into weights to prepare for color and transparency calculations. By weighting the color and transparency of each sample point and combining it with the coordinates in the canonical space, the expected surface intersection of the pixel can be calculated. Finally, by deforming the sample points and projecting them onto the camera, the final position of the pixel in the new image is calculated, and a 2D optical flow model is used to obtain the final volumetric rendering effect. Through this series of steps, the color and position changes of the original pixel can be accurately reconstructed in the new image frame.

[0041] The implementation of the various embodiments of this invention is based on programmed processing through a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of this invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of this invention provide a dynamic 3D model reconstruction system based on a high-frequency implicit network and a skinned model. This system is used to execute a dynamic 3D model reconstruction method based on a high-frequency implicit network and a skinned model from the above method embodiments.

[0042] The system comprises: a first main module for inputting multiple raw video segments and preprocessing them; a second main module for constructing a canonical shape model using a high-frequency implicit network and extracting density, canonical embedding, and color features from each preprocessed video frame; a third main module for inputting the extracted density, canonical embedding, and color features mapped to the canonical shape model into a neural hybrid skin deformation model for pose modeling and deformation control; a fourth main module for establishing a correspondence between pixels and a unified canonical 3D space in multi-frame image data and optimizing model parameters through canonical matching; and a fifth main module for volumetric rendering of the output features of the high-frequency implicit network and the neural hybrid skin deformation model, outputting the reconstructed dynamic 3D model.

[0043] This invention provides a dynamic 3D model reconstruction system based on a high-frequency implicit network and a skin model. Addressing the limitations of current methods in handling complex deformations and topological changes, and the difficulty in effectively capturing detailed information in dynamic scenes, this system employs several modules and combines a high-frequency implicit network and a skin model to improve the accuracy and stability of deformation representation. It can be applied to capture detailed features in videos for highly realistic dynamic 3D reconstruction.

[0044] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0045] Based on the same inventive concept as any of the foregoing embodiments, this embodiment of the invention also provides a dynamic 3D model reconstruction device based on a high-frequency implicit network and a skinned model, including a memory and a processor. The memory stores program instructions that are executed by the processor, and the processor calls the program instructions to execute the dynamic 3D model reconstruction method based on a high-frequency implicit network and a skinned model.

[0046] Based on the same inventive concept as any of the foregoing embodiments, this embodiment of the invention also provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute the dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models.

[0047] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0048] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0049] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0050] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0051] In summary, this invention discloses a dynamic 3D model reconstruction method and system based on a high-frequency implicit network and a skinning model. First, multiple original video segments are input and preprocessed. Then, an optimized high-frequency implicit network is used to model the geometric details of the input data, constructing a standardized shape model. Next, a neural hybrid skinning model is used to handle large-scale deformations and pose changes. Subsequently, a standardized matching module is used to achieve cross-viewpoint spatiotemporal consistency alignment. Finally, a high-fidelity dynamic 3D model is output through a volumetric rendering module. The neural hybrid skinning model includes a skeleton transformation module, a pose representation module, and a skinning weight module, combining an explicit Gaussian model with an implicit correction strategy to improve the accuracy and stability of deformation representation. This method can effectively capture detailed changes in dynamic scenes and has significant application value in fields such as video-based 3D reconstruction.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models, characterized in that, include: Input multiple raw video segments and perform preprocessing; A canonical shape model is constructed using a high-frequency implicit network, and density, canonical embedding, and color features are extracted from each preprocessed video frame. The extracted density, canonical embedding, and color features, which have been mapped to the canonical shape model, are input into the neural hybrid skin deformation model for pose modeling and deformation control. Establish the correspondence between pixels and a unified standard 3D space in multi-frame image data, and optimize model parameters through standard matching; Volumetric rendering is performed on the output features of the high-frequency implicit network and the neural hybrid skin deformation model to output the reconstructed dynamic 3D model.

2. The dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models according to claim 1, characterized in that, The preprocessing includes: Camera parameters are initialized, and the rendering range for each frame is dynamically adjusted according to the projection depth of the standard spatial surface. Root pose initialization: Train a CNN model, input the DensePose CSE feature map, predict the root pose for each frame, and align the target to the normal space. Pixel feature embedding uses the DensePose CSE model, a deep learning-based human pose estimation model, to extract pixel-level feature embeddings, providing semantic cues for dense correspondences across videos.

3. The dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models according to claim 1, characterized in that, Density, canonical embedding, and color features are extracted from each preprocessed video frame, including: Two concatenated quadratic convolutional layers are used to replace the linear layers in the SIREN network, and second-order constraint terms are introduced. An optimized SIREN network is used to perform density, canonical embedding, and color feature extraction on each video frame.

4. The dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models according to claim 1, characterized in that, Perform pose modeling and deformation control, including: Determining arbitrary 3D points using the skeleton transformation module Its corresponding 3D point Mapping relationships in the normative space; The root pose and body pose are modeled using a pose representation module. Use the skin weight module to generate skin weights for 3D skeletal animation.

5. The dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models according to claim 4, characterized in that, The skin weighting module also includes: The roughness weight module calculates roughness skin weights by explicitly modeling the local influence region of the skeleton using a 3D Gaussian ellipsoid. The detail correction module uses an optimized SIREN network to capture high-frequency, complex geometric details that are difficult to capture with Gaussian weights.

6. The dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models according to claim 1, characterized in that, Establishing a correspondence between pixels and a standardized 3D space in multi-frame image data includes: The pixel positions and features in each frame are mapped to a unified three-dimensional normed space and matched with the embeddings of other frames to achieve consistent alignment across viewpoints and time.

7. The dynamic 3D model reconstruction method based on high-frequency implicit networks and skinning models according to claim 1, characterized in that, Volumetric rendering is performed on the output features of the high-frequency implicit network and the neural hybrid skin deformation model, including: Given a pixel in one frame of an image, predict the color or position change of that pixel in another frame and reconstruct its position in 3D space.

8. A dynamic 3D model reconstruction system based on high-frequency implicit networks and skinning models, characterized in that, include: The first main module is used to input multiple raw video segments and perform preprocessing. The second main module is used to construct a canonical shape model using a high-frequency implicit network and extract density, canonical embedding, and color features from each preprocessed video frame. The third main module is used to input the extracted density, normalized embedding and color features that have been mapped to the normalized shape model into the neural hybrid skin deformation model for pose modeling and deformation control. The fourth main module is used to establish the correspondence between pixels and a unified standard three-dimensional space in multi-frame image data, and to optimize model parameters through standard matching. The fifth main module is used to perform volume rendering on the output features of the high-frequency implicit network and the neural hybrid skin deformation model, and output the reconstructed dynamic 3D model.

9. A dynamic 3D model reconstruction device based on a high-frequency implicit network and a skinned model, characterized in that, The system includes a memory and a processor, wherein the memory stores program instructions that are executed by the processor, and the processor invokes the program instructions to execute the dynamic 3D model reconstruction method based on a high-frequency implicit network and skinning model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the dynamic 3D model reconstruction method based on a high-frequency implicit network and skinning model as described in any one of claims 1 to 7.

Images