Digital human reconstruction method and system based on motion consistency constraint and computing equipment

Abstract

The invention provides a digital human reconstruction method and system based on motion consistency constraint and computing equipment, relates to the field of digital human reconstruction, and solves the technical problems of low reconstruction frame quality and jitter between adjacent frames in the digital human reconstruction process in the prior art. The method comprises the following steps: acquiring three-dimensional attitude parameters and body shape parameters of a figure in each frame in an input frame sequence, and acquiring features of a current frame through an image feature encoder; processing the three-dimensional attitude parameters and the body shape parameters based on a three-dimensional motion feature extraction algorithm to obtain three-dimensional motion features of the input frame sequence; processing the three-dimensional posture parameters based on a self-attention algorithm to obtain posture features of the human body; obtaining texture features based on the three-dimensional motion features and the features of the current frame; constructing a geometry through the posture characteristics of the human body; taking the geometry and the texture features as input of a three-dimensional Gaussian sputtering renderer, and outputting a digital human image frame under a specified viewpoint; the method and the device are used in a digital human reconstruction process.

Description

Technical Field

[0001] The present application relates to the field of digital human reconstruction, and in particular to a method, system, and computing device for digital human reconstruction based on motion consistency constraints. Background Art

[0002] Since its introduction, 3DGS (three-dimensional Gaussian sputtering) has quickly become a popular method for digital human reconstruction. However, current methods generally use pose as a constraint, but poses are sparse, resulting in limited supervision. Therefore, existing techniques consider adding motion constraints to the network's rendered output using the optical flow field between adjacent frames to achieve smooth rendering output. However, this 2D optical flow lacks sufficient spatial information, resulting in limited improvement. Another approach obtains temporal information by learning the deformation between adjacent frames. However, these methods lack explicit spatial constraints and therefore still cannot provide sufficient surface deformation supervision for rapidly changing human bodies. Consequently, the learned skinning function often exhibits jitter when driving the human body, resulting in inaccurate results.

[0003] The existing technology based on 3DGS still has difficulty in obtaining high-quality reconstruction in the process of further motion-driven regeneration, and has technical problems such as low reconstructed frame quality and jitter between adjacent frames.

[0004] Therefore, it is also necessary to study the feature expression method of digital human reconstruction and regeneration, and on this basis further design a new end-to-end high-fidelity digital reconstruction and regeneration method. Summary of the Invention

[0005] The present application provides a method, system and computing device for digital human reconstruction based on motion consistency constraints, which solves the technical problems of the existing technology based on 3DGS, which is still difficult to obtain high-quality reconstruction during the regeneration process driven by further action, and has the problems of low reconstructed frame quality and jitter between adjacent frames.

[0006] To achieve the above objectives, this application adopts the following technical solutions: In a first aspect, a digital human reconstruction method based on motion consistency constraints is provided, comprising: obtaining an input frame sequence; wherein the input frame sequence includes: a current frame and adjacent frames; The 3D pose parameters and body shape parameters of the character in each frame of the input frame sequence are obtained through multiple linear models based on skinning, and the features of the current frame are obtained through an image feature encoder; the 3D pose parameters and body shape parameters are processed based on a 3D motion feature extraction algorithm to obtain the 3D motion features of the input frame sequence; and the 3D pose parameters are processed based on a self-attention algorithm to obtain the posture features of the human body; Get texture features based on 3D motion features and current frame features; Construct geometric bodies based on human body posture features; The geometry and texture features are used as input to the 3D Gaussian splatter renderer, and the digital human image frame under the specified viewpoint is output.

[0007] In conjunction with the first aspect above, in one possible implementation, the adjacent frames include a frame before the current frame and a frame after the current frame; and the processing of the three-dimensional posture parameters and the body shape parameters based on the three-dimensional motion feature extraction algorithm to obtain the three-dimensional motion features of the input frame sequence includes: Retrieve the three-dimensional posture parameters marked as {M_θ}, the body shape parameters marked as {M_β}, and set the current frame The three-dimensional pose parameters and body shape parameters of the skinned multiple linear model (SMPL-X) are marked as ; The 3D pose parameters and body shape parameters of the previous frame are marked as , mark the three-dimensional posture parameters and body shape parameters of the next frame as ; By formula Calculate the current frame The corresponding velocity field of each Gaussian sphere ; By formula Calculate the current frame The corresponding acceleration field of each Gaussian sphere ;in, Indicates adjacent frames The corresponding velocity field of each Gaussian sphere; By formula Calculate the three-dimensional motion features of the input frame sequence ;in, Represents the embedding using the position of each Gaussian.

[0008] In conjunction with the first aspect above, in one possible implementation, the self-attention algorithm is used to process three-dimensional posture parameters to obtain posture features of the human body, including: The features of the 3D pose parameters are extracted through the global cross-attention mechanism to obtain the output features of the 3D pose of the character in adjacent frames; The features of the three-dimensional posture parameters are extracted through the local cross-attention mechanism to obtain the self-attention features of the three-dimensional posture of the character in the current frame; The output features are combined with the self-attention features to obtain the posture features of the human body.

[0009] In combination with the first aspect above, in one possible implementation, extracting features of three-dimensional pose parameters through a global cross-attention mechanism to obtain output features of the three-dimensional poses of the character in adjacent frames includes: Retrieve the three-dimensional posture of the character in the adjacent frame, and use the formula Calculate the output features ;in, Indicates based on and Cross-attention algorithm; Q represents the query vector matrix, K represents the key vector matrix, and V represents the value vector matrix. Indicates the selected frame among adjacent frames posture, Indicates the selected frame The poses of the adjacent frames; s=c±1.

[0010] It should be noted that the query vector matrix can be used instead of , the key vector matrix K replaces the second independent variable in the output feature , the value vector matrix V replaces the third independent variable in the output feature .

[0011] In combination with the first aspect above, in one possible implementation, extracting the features of the three-dimensional pose parameters through the local cross-attention mechanism to obtain the self-attention features of the three-dimensional pose of the person in the current frame includes: Retrieving the 3D posture of the character in the current frame, and dividing the 3D posture of the character into several posture parts according to the overall function; By formula Calculate the self-attention features of the current frame ;in, The i-th posture part representing the 3D posture of the character in the current frame; represents the rest of the posture parts except the i-th posture part.

[0012] In combination with the first aspect above, in a possible implementation, obtaining the texture feature based on the three-dimensional motion feature and the feature of the current frame includes: Input the features of the current frame and the 3D motion features into the geometric decoder to obtain the geometric features of the Gaussian sphere; wherein the geometric features of the Gaussian sphere include: the center position, orientation and radius of all Gaussian spheres of the human body; The geometric features of the Gaussian sphere and the features of the current frame are used as input to the texture decoder to obtain texture features.

[0013] In conjunction with the first aspect above, in one possible implementation, constructing a geometric body using posture features of a human body includes: The geometric features of the Gaussian sphere and the posture features of the human body are retrieved, and the geometric features of the Gaussian sphere and the posture features of the human body are used as independent variables of the linear mixed skinning function to output the geometry in the current posture.

[0014] In combination with the first aspect above, in a possible implementation, the output of the 3D Gaussian splatter renderer is subjected to reconstruction quality constraints based on a loss function; The loss function includes: color loss , Perceptual Loss , Laplace loss , optical flow loss ; By formula Calculate the total loss function ;in, Indicates that for the corresponding loss function The preset weight parameter; u is 、 、 or ; By formula The color loss between the color in the image frame rendered by 3DGS and the true value is calculated; where R represents the color value of each pixel in the current frame, Represents the true color value of each pixel in the image frame at a specified viewpoint; By formula The perceptual loss between the feature map of the image frame rendered by 3DGS and the corresponding true value is calculated; is the feature map of the current frame, The true value of the feature map of the image frame rendered by 3DGS; This is the operation mode of the convolutional neural network VGG; By formula Calculate the Laplace loss ;in, represents the properties of the predicted i-th Gaussian ball, Represents the true value of the attribute of the i-th Gaussian ball; By formula Calculate the optical flow loss ;in, represents the optical flow function.

[0015] It should be noted that Expressed by the L1 norm method, 、 as well as It is expressed by the L2 norm method.

[0016] In a second aspect, the present application provides a digital human reconstruction computing device based on motion consistency constraints, comprising: a processor and a storage medium; the storage medium comprises instructions, and the processor is used to execute the instructions to implement the method described in the first aspect and any possible implementation of the first aspect.

[0017] In a third aspect, the present application provides a digital human reconstruction system based on motion consistency constraints, comprising: a data acquisition module, a feature extraction module and a constraint reconstruction module; The data acquisition module is used to obtain an input frame sequence, wherein the input frame sequence includes a current frame and adjacent frames. The three-dimensional pose parameters and body shape parameters of the character in each frame of the input frame sequence are obtained through multiple linear models based on skinning, and the features of the current frame are obtained through an image feature encoder; The feature extraction module is used to: process 3D posture parameters and body shape parameters based on a 3D motion feature extraction algorithm to obtain 3D motion features of the input frame sequence; and process 3D posture parameters based on a self-attention algorithm to obtain posture features of the human body; The constrained reconstruction module is used to: obtain texture features based on 3D motion features and features of the current frame; construct geometric bodies based on human posture features; The geometry and texture features are used as input to the 3D Gaussian splatter renderer, and the digital human image frame under the specified viewpoint is output.

[0018] The present application provides a digital human reconstruction method, system and computing device based on motion consistency constraints, which can use the motion field composed of adjacent three-dimensional surface velocities and accelerations as a constraint to obtain rich surface motion information, overcome the description limitations of traditional two-dimensional optical flow fields or sparse posture expressions on three-dimensional surfaces, and help achieve more accurate digital human reconstruction; secondly, it introduces a global and local two-level attention mechanism to realize the posture feature extraction of the current posture from the adjacent posture under the action of global cross-attention and local cross-attention, obtain smoother and higher-fidelity posture features, thereby further serving the skin function estimation and action-driven regeneration, thereby achieving high-quality feature expression of the digital human in the reconstruction process.

[0019] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of a technical feature, technical solution or beneficial effect in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the module relationship of a digital human reconstruction system based on motion consistency constraints provided in an embodiment of the present application; Figure 2 A flowchart of a digital human reconstruction method based on motion consistency constraints provided in an embodiment of the present application; Figure 3 A flowchart of another method for reconstructing a digital human based on motion consistency constraints provided in an embodiment of the present application; Figure 4 A flowchart of another method for reconstructing a digital human with motion consistency constraints provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "a plurality" means two or more. Words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not limit them to be different.

[0022] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0023] The digital human reconstruction method based on motion consistency constraint provided in the embodiment of the present application can be applied to Figure 1-Figure 2 In the digital human reconstruction system based on motion consistency constraints shown, the communication system includes: a data acquisition module, a feature extraction module and a constraint reconstruction module; Among them, the data acquisition module includes: SMPL-X unit; The feature extraction module includes: a three-dimensional motion feature extraction unit and a posture feature extraction unit; The constrained reconstruction module includes: geometry decoder, implicit LBS learning unit, texture decoder and 3D Gaussian sputtering renderer; The SMPL-X unit is used to call multiple skin-based linear models and use them as a highly realistic parametric model of the human body. This model can represent more detailed structural changes in the body, face, and hands, providing structural priors for subsequent high-precision reconstruction. The 3D motion feature extraction unit is used to call the 3D motion feature extraction algorithm. It consists of a multi-layer perceptron (MLP). Each layer contains a fully connected layer with an input dimension of 512 and an output dimension of 256. The activation function is ReLU. The last layer outputs a 128-dimensional 3D motion feature vector f3D. This module integrates the velocity and acceleration fields of the Gaussian ball in multiple time frames to achieve temporal dynamic modeling, capture the changes in motion trends in the time domain, and enhance the dynamic perception ability of the current frame. The posture feature extraction unit uses a self-attention algorithm based on a two-layer attention mechanism. It consists of a local attention layer (using a two-head multi-head attention with a hidden dimension of 256) and a global attention layer (four heads with a hidden dimension of 512). These layers process the posture alignment between local adjacent frames and the global semantic consistency between frames, respectively, to extract structure-aware human posture features. The human posture feature dimension is set to 256. Furthermore, the image features of the current frame (extracted by a ResNet-18 backbone network with a 512-dimensional output) are concatenated and fused with the 3D motion features (128-dimensional) in the channel dimension to form a 640-dimensional feature, which serves as the input to the geometry decoder. The geometric decoder consists of a four-layer fully connected network. Each layer outputs 512, 256, 128, and the final Gaussian attribute parameter vector (the center position, orientation, and radius of all Gaussian spheres on the human body). The center position vector has a dimension of 3, the orientation is represented by a quaternion as 4, and the radius is 3, for a total output of 10 dimensions. Each layer uses the ReLU activation function, and the last layer is a linear layer. The geometric attribute vector is input into the implicit LBS learning module together with the posture features. Implicit LBS learning unit, used to call the linear hybrid skinning function. This module is based on the MLP structure and adopts a three-layer design (input is 256 dimensions, and output is the skin weight distribution vector corresponding to each Gaussian sphere). It is used to learn the mapping relationship from Gaussian point-driven skeleton motion to body surface deformation, supporting high-quality pose deformation expression; The texture decoder is used to concatenate the geometric features obtained from the geometry decoder with the texture features extracted from the image (extracted by the UNet structure and ultimately compressed to 256 dimensions) as the texture decoder input. The texture decoder consists of a three-layer fully connected structure that outputs color (RGB, 3D) and transparency (1D) respectively, and ultimately outputs them as a 4D RGBA vector to the 3DGS renderer (i.e., 3D Gaussian splatter renderer). The 3D Gaussian splattering renderer is used to achieve end-to-end training based on multi-view image reconstruction supervision through differentiable rendering of multiple-view images, thereby ensuring the dual consistency of geometric structure restoration and visual texture reconstruction, and realizing high-fidelity, high-consistency, multi-view digital human reconstruction and regeneration.

[0024] To address the technical issues of the existing 3DGS-based methods in the process of further motion-driven regeneration, which still result in difficulty in obtaining high-quality reconstruction, low reconstructed frame quality, and jitter between adjacent frames, the present invention provides a digital human reconstruction method based on motion consistency constraints, the method comprising: obtaining an input frame sequence; wherein the input frame sequence comprises: a current frame and adjacent frames; The 3D pose parameters and body shape parameters of the character in each frame of the input frame sequence are obtained through multiple linear models based on skinning, and the features of the current frame are obtained through an image feature encoder; the 3D pose parameters and body shape parameters are processed based on a 3D motion feature extraction algorithm to obtain the 3D motion features of the input frame sequence; and the 3D pose parameters are processed based on a self-attention algorithm to obtain the posture features of the human body; Get texture features based on 3D motion features and current frame features; Construct geometric bodies based on human body posture features; The geometric and texture features are used as inputs of a three-dimensional Gaussian splatter renderer, and a digital human image frame at a specified viewpoint is output; based on this, the present invention achieves high-quality feature expression of the digital human during the reconstruction process.

[0025] like Figure 3 As shown, the digital human reconstruction method based on motion consistency constraint provided by the embodiment of the present application includes: Based on the 3D motion feature extraction algorithm, the 3D posture parameters and body shape parameters are processed to obtain the 3D motion features of the input frame sequence; In some implementations, an input frame sequence is obtained, and three-dimensional pose parameters and body shape parameters of a character in each frame of the input frame sequence are obtained through multiple linear models based on skinning; wherein the input frame sequence includes: a current frame and adjacent frames; the three-dimensional pose parameters are retrieved and labeled as {M_θ}, and the body shape parameters are labeled as {M_β}, and the three-dimensional pose parameters and body shape parameters of the current frame based on SMPL-X are labeled as ; The 3D pose parameters and body shape parameters of the previous frame are marked as , mark the three-dimensional posture parameters and body shape parameters of the next frame as ; By formula Calculate the current frame The corresponding velocity field of each Gaussian sphere ; By formula Calculate the current frame The corresponding acceleration field of each Gaussian sphere ; By formula Calculate the three-dimensional motion features of the input frame sequence ;in, Represents the embedding using the position of each Gaussian.

[0026] It should be pointed out that An implicit calculation method for a three-dimensional motion encoder.

[0027] For example, we now obtain the input frame sequence 、 as well as ,Will As the current frame, and Marked as The adjacent frame is now set 、 and is the same vertex in the three input frame sequences. The three-dimensional pose parameters {M_θ} and body shape parameters {M_β} of the character in each frame of the input frame sequence are obtained through multiple linear models based on skinning, and the three-dimensional pose parameters and body shape parameters of the current frame based on SMPL-X are marked as ; The three-dimensional pose parameters and body shape parameters of adjacent frames based on SMPL-X are marked as and ; for , initialize a Gaussian sphere for each point on the SMPL-X surface point to obtain a standard surface based on 3DGS ; Among them, the Gaussian sphere is represented by the Gaussian function, and the radius of the Gaussian sphere is set to 1, the rotation (orientation) is the unit matrix, and the opacity is set to 1, thereby completing the Gaussian sphere initialization; By formula Calculate the current frame The corresponding velocity field of each Gaussian sphere ; By formula Calculate the current frame The corresponding acceleration field of each Gaussian sphere ; By formula Calculate the three-dimensional motion features of the input frame sequence .

[0028] like Figure 4 As shown, the digital human reconstruction method based on motion consistency constraint provided by the embodiment of the present application includes: Based on the self-attention algorithm, the three-dimensional posture parameters are processed to obtain the posture characteristics of the human body; In some implementations, the 3D poses of the characters in adjacent frames are retrieved and the equation Calculate the output features ;in, Indicates based on and Cross-attention algorithm; Q represents the query vector matrix, K represents the key vector matrix, and V represents the value vector matrix. represents the pose of the selected frame s among the adjacent frames, represents the pose of the adjacent frames of the selected frame s; s=c±1; Retrieving the 3D posture of the character in the current frame, and dividing the 3D posture of the character into several posture parts according to the overall function; By formula Calculate the self-attention features of the current frame ;in, The i-th posture part representing the 3D posture of the character in the current frame; represents the rest of the posture parts except the i-th posture part; The output features and self-attention features are combined to obtain the posture features of the human body .

[0029] For example, the 3D posture of the character is retrieved and first enters the global cross attention calculation module through the formula get ;in, Indicates that Cross attention calculation; Similarly, for the current frame, its self-attention feature based on its own posture can also be calculated Here, in order to better calculate the posture attention, the human body can be divided into five parts according to the overall function, including the left arm, right arm, left leg, right leg and torso, and then the partial blocks are cross-attended and then fused to obtain the final , Right now: ;in, Represents the i-th part of the 5 posture parts of the current frame; Indicates the remaining part of the 5 pose parts of the current frame.

[0030] The digital human reconstruction method based on motion consistency constraint provided in the embodiment of the present application includes: constructing a loss function to constrain the reconstruction quality of the output of a three-dimensional Gaussian splatter renderer; In some implementations, the loss function includes: color loss , Perceptual Loss , Laplace loss , optical flow loss ; By formula Calculate the total loss function ;in, Indicates that for the corresponding loss function The preset weight parameter; u is 、 、 or ; For example, since constructing the loss function is a self-supervised process, both training and testing can be carried out using the ZJU-MoCap dataset and the real-shot dataset; the input video length can be 1 second or longer (for example, 30 frames or 60 frames), the frame rate is set to 25fps or 30fps, and the resolution can be 512×512 or 1024×1024, or even larger high-resolution images to obtain clearer texture and posture features (the richer the perspective, the higher the quality of reconstruction and generation). During training, you can try to change the weight parameters and learning rate strategy in multiple stages (the initial learning rate is , and then cosine annealing is used to decay to ) and other combinations have clearer texture and posture features; Among them, the weight parameter 、 、 and How to obtain: Step 1: Set the initial weight parameters (referred to as group A) so that the frame rate of the image frame reaches the initial target frame rate, and input the weight parameters into the corresponding loss function to obtain the initial total loss function value; Step 2: Replace a set of initial weight parameters (referred to as group B) with the goal of obtaining higher quality image frames, input the replaced weight parameters into the corresponding loss function to obtain the second total loss function value, and calculate the difference between the absolute values ​​of the initial total loss function value and the second total loss function value as the first descent rate; Step 3: Repeat step 2 to obtain a new set of weight parameters (referred to as group C) and input them into the corresponding loss function to obtain the third total loss function value, and calculate the difference between the absolute value of the second total loss function value and the second total loss function value, which is the second decline rate; Step 4: Determine whether the first drop rate is greater than the second drop rate; if so, use the weight parameter of group B as 、 、 and If not, the weight parameter of group C is used as 、 、 and The corresponding value of .

[0031] This invention addresses the problems that existing 3DGS-based methods cannot reconstruct high-quality digital humans and cannot regenerate high-quality digital humans under action-driven conditions. First, it proposes to achieve high-quality feature expression from two perspectives; then, it further combines the two expressions to propose an end-to-end digital human reconstruction and regeneration framework; In the system framework, we embed the above high-quality feature expressions into an end-to-end digital human reconstruction and regeneration, and combine these two high-quality features with skinning function learning, geometric reconstruction and texture estimation to achieve higher-fidelity 3DGS-based digital human reconstruction and posture-based motion-driven regeneration than previous work.

[0032] The above mainly introduces the scheme of the embodiment of the present application from the perspective of device implementation. It is understandable that each device, for example, each device in the digital human reconstruction system based on motion consistency constraints, contains at least one of the hardware structure and software modules corresponding to the execution of each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0033] The embodiment of the present application can divide the digital human reconstruction system based on motion consistency constraints into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0034] During implementation, each step of the method provided in this embodiment can be completed by hardware integrated logic circuits in a processor or by software instructions. The steps of the method disclosed in the embodiments of this application can be directly implemented as execution by a hardware processor, or as a combination of hardware and software modules in a processor.

[0035] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, among other types of computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform operations or processing. The processor may be a separate semiconductor chip or integrated into a semiconductor chip with other circuits. For example, it may form an SoC (system on a chip) with other circuits (such as a codec circuit, a hardware acceleration circuit, or various bus and interface circuits). Alternatively, it may be integrated into an ASIC as a built-in processor. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the core for executing software instructions to perform operations or processing, the processor may further include necessary hardware accelerators, such as a field programmable gate array (FPGA), a PLD (programmable logic device), or logic circuits that implement specialized logic operations.

[0036] The memory in the embodiments of the present application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, and electrically erasable programmable read-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0037] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0038] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. The fact that certain measures are recorded in different dependent claims does not mean that these measures cannot be combined to produce good results.

[0039] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

Claims

1. A digital human reconstruction method based on motion consistency constraints, characterized by: include: Obtain an input frame sequence; wherein the input frame sequence includes: a current frame and adjacent frames; The 3D pose parameters and body shape parameters of the character in each frame of the input frame sequence are obtained through multiple linear models based on skinning, and the features of the current frame are obtained through an image feature encoder; the 3D pose parameters and body shape parameters are processed based on a 3D motion feature extraction algorithm to obtain the 3D motion features of the input frame sequence; and the 3D pose parameters are processed based on a self-attention algorithm to obtain the posture features of the human body; Get texture features based on 3D motion features and current frame features; Construct geometric bodies based on human body posture features; The geometry and texture features are used as input to the 3D Gaussian splatter renderer, and the digital human image frame under the specified viewpoint is output.

2. The digital human reconstruction method based on motion consistency constraint according to claim 1, characterized in that: The adjacent frames include a previous frame of the current frame and a next frame of the current frame; The three-dimensional motion feature extraction algorithm is used to process the three-dimensional posture parameters and body shape parameters to obtain the three-dimensional motion features of the input frame sequence, including: The three-dimensional posture parameters are marked as {M_θ}, the body shape parameters are marked as {M_β}, and the three-dimensional posture parameters and body shape parameters of the current frame are marked as ; The 3D pose parameters and body shape parameters of the previous frame are marked as , mark the three-dimensional posture parameters and body shape parameters of the next frame as ; By formula Calculate the velocity field of each Gaussian ball corresponding to the current frame ; By formula Calculate the current frame The corresponding acceleration field of each Gaussian sphere ; By formula Calculate the three-dimensional motion features of the input frame sequence ;in, Represents the embedding using the position of each Gaussian.

3. The digital human reconstruction method based on motion consistency constraint according to claim 1, characterized in that: The self-attention algorithm is used to process the three-dimensional posture parameters to obtain the posture features of the human body, including: The features of the 3D pose parameters are extracted through the global cross-attention mechanism to obtain the output features of the 3D pose of the character in adjacent frames; The features of the three-dimensional posture parameters are extracted through the local cross-attention mechanism to obtain the self-attention features of the three-dimensional posture of the character in the current frame; The output features are combined with the self-attention features to obtain the posture features of the human body.

4. The digital human reconstruction method based on motion consistency constraint according to claim 3 is characterized in that: The method of extracting the features of the three-dimensional posture parameters through the global cross attention mechanism to obtain the output features of the three-dimensional posture of the character in adjacent frames includes: By formula Calculate the output features ; in, Indicates based on Cross-attention algorithm, represents the pose of the selected frame among the adjacent frames, Indicates the poses of the frames adjacent to the selected frame.

5. The digital human reconstruction method based on motion consistency constraint according to claim 3 is characterized in that: The feature extraction of the three-dimensional posture parameters by the local cross-attention mechanism to obtain the self-attention feature of the three-dimensional posture of the character in the current frame includes: Divide the character's three-dimensional posture into several posture parts according to their overall functions; By formula Calculate the self-attention features of the current frame ;in, The i-th posture part representing the 3D posture of the character in the current frame; represents the rest of the posture parts except the i-th posture part.

6. The digital human reconstruction method based on motion consistency constraint according to claim 1, characterized in that: The obtaining of texture features based on the three-dimensional motion features and the features of the current frame includes: Input the features of the current frame and the 3D motion features into the geometric decoder to obtain the geometric features of the Gaussian sphere; wherein the geometric features of the Gaussian sphere include: the center position, orientation and radius of all Gaussian spheres of the human body; The geometric features of the Gaussian sphere and the features of the current frame are used as input to the texture decoder to obtain texture features.

7. The method for digital human reconstruction based on motion consistency constraint according to claim 6, characterized in that: The method of constructing a geometric body based on the posture characteristics of the human body includes: The geometric features of the Gaussian sphere and the posture features of the human body are retrieved, and the geometric features of the Gaussian sphere and the posture features of the human body are used as independent variables of the linear mixed skinning function to output the geometry in the current posture.

8. The digital human reconstruction method based on motion consistency constraint according to claim 1, characterized in that: The output of the three-dimensional Gaussian splatter renderer is reconstructed with quality constraints based on a loss function; The loss function includes: color loss , Perceptual Loss , Laplace loss , optical flow loss ; By formula Calculate the total loss function ;in, Indicates that for the corresponding loss function The preset weight parameter; u is 、 、 or ; By formula The color loss between the color in the image frame rendered by 3DGS and the true value is calculated; where R represents the color value of each pixel in the current frame, Represents the true color value of each pixel in the image frame at a specified viewpoint; By formula The perceptual loss between the feature map of the image frame rendered by 3DGS and the corresponding true value is calculated; is the feature map of the current frame, The true value of the feature map of the image frame rendered by 3DGS; This is the operation mode of the convolutional neural network VGG; By formula Calculate the Laplace loss ;in, represents the properties of the predicted i-th Gaussian ball, Represents the true value of the attribute of the i-th Gaussian ball; By formula Calculate the optical flow loss ;in, represents the optical flow function.

9. A digital human reconstruction computing device based on motion consistency constraints, characterized in that: include: A memory and a processor, wherein the memory stores executable instructions of the processor; wherein the processor is configured to execute the digital human reconstruction method based on motion consistency constraints according to any one of claims 1 to 8 by executing the executable instructions.

10. A digital human reconstruction system based on motion consistency constraints, adapted to the digital human reconstruction method based on motion consistency constraints according to any one of claims 1 to 8, characterized in that: include: Data acquisition module, feature extraction module and constraint reconstruction module; The data acquisition module is used to: obtain an input frame sequence, obtain the three-dimensional posture parameters and body shape parameters of the character in each frame of the input frame sequence through multiple linear models based on skinning; obtain the features of the current frame through an image feature encoder; wherein the input frame sequence includes: the current frame and adjacent frames; The feature extraction module is used to obtain the 3D motion features of the input frame sequence based on the 3D posture parameters and body shape parameters as inputs to the 3D motion feature extraction algorithm; and obtain the posture features of the human body through the self-attention algorithm based on the 3D posture parameters. The constrained reconstruction module is used to: obtain texture features through three-dimensional motion features and features of the current frame; construct geometric bodies through human posture features; use geometric bodies and texture features as input of a three-dimensional Gaussian sputtering renderer, output image frames under a specified viewpoint through the three-dimensional Gaussian sputtering renderer, and reconstruct quality constraints on the output of the three-dimensional Gaussian sputtering renderer by constructing a loss function.

Images