Digital human-driven key point generation method and device, and medium

Abstract

The invention discloses a digital human-driven key point generation method and device and a medium, and the method comprises the steps: extracting a static spatial domain feature and a dynamic time domain feature of a standardized target voice through a constructed spatial domain-time domain double-flow feature decoupling model; according to a cross-modal dynamic attention fusion mechanism, performing dynamic weighted fusion on the static spatial domain features and the dynamic time domain features to obtain fusion features; according to a biomechanical constraint algorithm, trajectory operation is carried out on the fusion features, and a key point motion trajectory conforming to biomechanical constraints is generated; and through a joint loss function, calibrating the key point motion trail and the key point trail of the standardized target video, and generating a final face key point. According to the scheme, the synchronization precision of the digital human lip shape is improved by constructing a space-time domain double-flow feature decoupling architecture; the non-physical motion is eliminated through a biomechanical constraint algorithm, the track smoothness is improved, the phoneme and lip-shaped semantic association ability is enhanced through cross-modal fusion, and the generalization of roles is ensured.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to a method, device and medium for generating key points for digital human-driven systems. Background Technology

[0002] Digital human-driven refers to the technical means that enable digital humans to exhibit dynamic behaviors, expressions, and language, and to interact with the environment or users.

[0003] The existing digital human-driven approach has the following key defects: When the key points of the parameter-driven general model are generated across roles, the retention rate of role identity features is less than 60% due to the different facial adaptations of each role, which greatly reduces the role generalization ability.

[0004] The improved Mel spectral feature method, at the temporal modeling level, often uses a single LSTM or GRU network to process audio features. The lip movement trajectory has non-physical jitter and cannot capture the deep semantic relationship between phonemes and facial muscle movements, resulting in a significant reduction in lip synchronization accuracy and unreasonable digital human movement trajectory. Summary of the Invention

[0005] This application provides a method, device, and medium for generating key points for digital human driving, which addresses the problems of existing digital human driving methods in terms of lip-sync accuracy, motion trajectory rationality, and cross-role generalization ability.

[0006] The embodiments of this application adopt the following technical solutions: On one hand, embodiments of this application provide a digital human-driven keypoint generation method, which includes: extracting static spatial features and dynamic temporal features of standardized target speech through a pre-constructed spatial-temporal dual-stream feature decoupling model; dynamically weighting and fusing the static spatial features and dynamic temporal features according to a cross-modal dynamic attention fusion mechanism to obtain fused features; performing trajectory calculation on the fused features according to a biomechanical constraint algorithm to generate keypoint motion trajectories that conform to biomechanical constraints; and calibrating the keypoint motion trajectories with the keypoint trajectories of the standardized target video through a joint loss function to generate the final facial keypoints.

[0007] In one example, the motion trajectory of the keypoints is calibrated with the keypoint trajectory of the standardized target video using a joint loss function to generate the final facial keypoints. Specifically, this includes: calculating the error between the motion trajectory of the keypoints and the trajectory of the keypoints in the standardized target video using an L1 loss function to calibrate the motion trajectory of the keypoints; calibrating the motion trajectory of keypoints outside a preset jitter acceleration range using an acceleration smoothing loss function to ensure that the jitter acceleration of the keypoint motion trajectories falls within the preset jitter acceleration range; calibrating the motion trajectory of keypoints outside a preset lip movement acceleration range using a physical constraint loss function to ensure that the lip movement acceleration of the keypoint motion trajectories falls within the preset lip movement acceleration range; and determining the keypoints of the calibrated keypoint motion trajectory as the final facial keypoints.

[0008] In one example, a biomechanical constraint algorithm is used to perform trajectory calculations on the fused features to generate keypoint motion trajectories that conform to biomechanical constraints. Specifically, this includes: constructing the second derivative covariance matrix of adjacent keypoints of the fused features based on the Gaussian process regression equation; optimizing the second derivative covariance matrix using the Lagrange algorithm based on preset jitter acceleration and coordinate accuracy weights to obtain the standard covariance matrix; solving the standard covariance matrix using Cholesky decomposition to obtain the primary keypoint motion trajectory; and eliminating non-physical abrupt changes in the primary keypoint motion trajectory through physical constraints to generate keypoint motion trajectories that conform to biomechanical constraints.

[0009] In one example, based on a cross-modal dynamic attention fusion mechanism, the static spatial features and the dynamic temporal features are dynamically weighted and fused to obtain fused features. Specifically, this includes: fusing the static spatial features and the dynamic temporal features through an attention weight matrix to obtain initial fused features; using the processing result of the 8-head self-attention mechanism on the static spatial features as the key and the processing result of the 8-head self-attention mechanism on the dynamic temporal features as the value; superimposing the initial fused features with the static spatial features and the dynamic temporal features respectively through residual connections, and using the key and value as features for feature filtering respectively to obtain fused features.

[0010] In one example, before extracting the static spatial features and dynamic temporal features of the target speech using a pre-built spatial-temporal dual-stream feature decoupling model, the method further includes: determining the target speech according to a continuous speech sampling protocol and a preset speech sampling frequency; determining the target video according to preset camera parameters and ambient brightness; and aligning the timestamps of the target speech and the target video according to the PTP protocol to achieve audio sampling synchronization.

[0011] In one example, after aligning the target speech and target video with timestamps according to the PTP protocol to achieve audio sampling synchronization, the method further includes: increasing the signal-to-noise ratio of the target speech to a preset signal-to-noise ratio using the RNNoise algorithm; extracting features from the increased target speech to obtain the third-level MFCC features of the target speech; annotating the increased speech with phonemes using the ConformerASR system to obtain phoneme annotation results; integrating the third-level MFCC features and the phoneme annotation results into a T×26-dimensional temporal feature matrix and outputting it in NPZ format to obtain the standardized target speech; where T is the audio length.

[0012] In one example, after aligning the target speech and the target video with timestamps according to the PTP protocol to achieve audio sampling synchronization, the method further includes: performing face detection on the target video using a MediaPipe facial mesh to determine facial key points; performing feature normalization processing on the facial key points to obtain normalized features; performing temporal domain enhancement, spatial domain enhancement, and adversarial example enhancement on the target video to obtain multi-dimensional enhanced data; integrating the normalized features and the multi-dimensional enhanced data to generate a T×68×2 dimensional NPZ file to obtain the normalized target video; where T is the number of video frames.

[0013] In one example, before extracting the static spatial features and dynamic temporal features of the standardized target speech using a pre-built spatial-temporal dual-stream feature decoupling model, the method further includes: constructing a spatial-temporal dual-stream feature decoupling model; the spatial feature stream of the spatial-temporal dual-stream feature decoupling model adopts a ResNet-50 architecture, including 3 layers of 1D convolution and max pooling, outputting a 256-dimensional MFCC feature vector; the temporal feature stream of the spatial-temporal dual-stream feature decoupling model adopts a TCN architecture, including 4 sets of causal convolutional layers, each equipped with a gated activation unit (GLU).

[0014] On the other hand, embodiments of this application provide a digital human-driven keypoint generation device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned digital human-driven keypoint generation methods.

[0015] On the other hand, embodiments of this application provide a non-volatile computer storage medium for generating digital human-driven key points, which stores computer-executable instructions capable of executing any of the above-mentioned digital human-driven key point generation methods.

[0016] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: This solution constructs a spatial-temporal dual-stream feature decoupling architecture, reducing the lip-sync error (LSE-D) from the baseline of 4.1mm to 2.8mm, a reduction of 31.7%. This effectively solves the problem that traditional models cannot simultaneously process audio spatial and temporal features, improving the temporal alignment accuracy of lip movements and speech. By eliminating non-physical motion through a biomechanical constraint algorithm, the smoothness of keypoint trajectories (SPL index) is improved by 2.3 times (from 0.45 to 1.03), and the acceleration of plosive and fricative transition segments is reduced by 42%, with motion trajectories conforming to human biomechanical principles. Cross-modal fusion enhances the semantic association between phonemes and lip movements, achieving a 92.4% identity retention rate in Few-shot scenarios (compared to 73.5% at baseline), significantly improving cross-role generalization ability. Attention mechanisms focus on key articulation regions, effectively capturing the deep semantic association between phonemes and facial muscle movements, and the expressive power of MFCC features overcomes the limitations of "hand-designed" approaches. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which: Figure 1 A flowchart illustrating a digital human-driven keypoint generation method provided in an embodiment of this application; Figure 2 A spatiotemporal feature decoupling model framework diagram of a digital human-driven keypoint generation method provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a digital human-driven keypoint generation device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a digital human-driven keypoint generation method provided in an embodiment of this application. This method can be applied to different business domains. Certain input parameters or intermediate results in this process can be manually adjusted to help improve accuracy.

[0021] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a controller as an example.

[0022] Based on this Figure 1 The process may include the following steps: S101: Extract static spatial features and dynamic temporal features of standardized target speech by using a pre-constructed spatial-temporal dual-stream feature decoupling model.

[0023] In some embodiments of this application, before extracting the static spatial features and dynamic temporal features of the standardized target speech using a pre-built spatial-temporal dual-stream feature decoupling model, it is necessary to first acquire the target speech and the target video.

[0024] First, regarding the acquisition of the target speech: sound was collected in an acoustic processing room (noise floor <20dB) using a Neumann U87 microphone (24bit / 48kHz sampling). Continuous recording (5-10 seconds per segment) was performed according to the continuous speech sampling protocol and preset speech sampling frequency, including normal speech speed (150 words / minute), fast speech speed (250 words / minute), and emotional speech (6 states such as anger and joy), which were then identified as the target speech.

[0025] Secondly, regarding the acquisition of the target video: it was recorded using a Sony FX6 4K camera (120fps@1080p mode) under a constant light source (illuminance adjustable from 500-1000 lux), f / 2.8 aperture, ISO 200, and automatic white balance lock, and was identified as the target video.

[0026] Finally, after determining the target speech and target video, the target speech and target video are timestamped according to the PTP protocol to achieve audio sampling synchronization (error <1ms).

[0027] Furthermore, after audio synchronization, the target video and target audio need to be preprocessed. Specific steps include: The signal-to-noise ratio (SNR) of the target speech is enhanced to a preset SNR (e.g., SNR ≥ 15dB) using the RNNoise algorithm. Then, feature extraction is performed on the enhanced target speech to obtain three levels of MFCC features: fundamental frequency feature: 13-dimensional MFCC (Δ = 25ms), energy feature: 13-dimensional short-time energy (Δ = 10ms), and dynamic feature: 13-dimensional second-order difference (Δ = 5ms). Finally, the enhanced speech is phoneme-annotated using the Conformer ASR system to obtain phoneme annotation results (annotation error rate < 3%). Finally, the level 3 MFCC features and the phoneme annotation results are integrated into a T×26-dimensional temporal feature matrix and output in NPZ format to obtain the standardized target speech; where T is the audio length.

[0028] Simultaneously, face detection is performed on the target video using MediaPipe facial mesh to determine facial key points (68-point detection, mAP = 98.7%). Then, the facial key points are subjected to feature standardization processing to obtain standardized features. Specifically, this includes coordinate normalization and motion decomposition, such as mapping the key point coordinates to the [0,1] space (reference: interpupillary distance = 1 unit) and separating the static reference shape (S) from the dynamic deformation (D = FS).

[0029] Then, temporal enhancement, spatial enhancement, and adversarial example enhancement are performed on the target video to obtain multi-dimensional enhanced data. Finally, the standardized features and multi-dimensional enhanced data are integrated to generate a T×68×2 dimensional NPZ file to obtain the standardized target video; where T is the number of video frames.

[0030] Furthermore, an improved Dynamic Time Warping (DTW) algorithm is used to achieve audio-video alignment, with the following formula: ; in, The window size constraint is ±120ms.

[0031] Furthermore, after achieving audio-video alignment, it is necessary to construct a spatial-temporal dual-stream feature decoupling model. The spatial feature stream of the spatial-temporal dual-stream feature decoupling model adopts the ResNet-50 architecture, including a 1×T×26 MFCC feature tensor (T is the audio length), a feature encoding module, and a feature decoupling module.

[0032] The feature encoding module includes: Convolutional layer 1: Conv1D(256,7,2), BatchNorm, ReLU; Convolutional layer 2: Conv1D(256,3,1), DepthwiseConv1D(256,3,1); Residual block: containing 3 Bottleneck structures (64 to 128 to 256 channels); Frequency band separation module: channel attention mechanism (separating frequency band importance weights); Fundamental frequency enhancement module: 1×1 convolution to extract fundamental frequency features (32 channels); Lip feature module: 3×1 convolution to extract lip-related frequency bands (128 channels).

[0033] The feature decoupling module includes: static feature branch: global average pooling, downsampling from 256 to 128 channels; dynamic feature branch: temporal convolutional layer (3×1 convolution, stride 2); feature fusion: cross-layer feature fusion is achieved through skip connections (skip stride 2).

[0034] Furthermore, the temporal feature stream of the spatial-temporal dual-stream feature decoupling model adopts the TCN architecture, which includes four causal convolutional layers, each equipped with a gated activation unit (GLU).

[0035] After constructing the spatial-temporal dual-stream feature decoupling model, the static spatial features and dynamic temporal features of the standardized target speech are extracted.

[0036] It should also be noted that in some embodiments of this application, the dataset for model training adopts a three-level dataset system, specifically including: a basic training set (VoxCeleb2 enhanced version), with a sample size of 1000 speakers, 50 videos per speaker (a total of 50,000 samples). The feature dimensions are MFCC (26-dimensional), HuBERT (768-dimensional), and keypoints (68×2-dimensional), and are manually verified twice (keypoint error <0.5mm).

[0037] The Few-Shot test set includes 10 target characters, with one 10-second video (for training) and five test videos for each character.

[0038] Furthermore, data augmentation strategies were employed, specifically: 100 virtual samples were generated by 3D deformation field interpolation (based on the 3DMM model); Gaussian noise (σ=0.1mm) was added and motion blending was performed for cross-role keypoint motion transfer (preserving topology).

[0039] Finally, a cross-language validation set, including Chinese and English datasets, was used to establish a cross-language mapping based on a phoneme correspondence table (CMU phoneme library).

[0040] By constructing a spatial-temporal dual-stream feature decoupling architecture, the lip-sync accuracy of digital humans was greatly improved, and the lip-sync error (LSE-D) was reduced from the baseline of 4.1 mm to 2.8 mm, a reduction of 31.7%.

[0041] S102: Based on the cross-modal dynamic attention fusion mechanism, the static spatial features and the dynamic temporal features are dynamically weighted and fused to obtain fused features.

[0042] In some embodiments of this application, after extracting the static spatial features and dynamic temporal features of the standardized target speech, it is necessary to dynamically weight and fuse the static spatial features and dynamic temporal features. Before performing the weighted fusion, it should be noted that this application achieves dynamic weighted fusion of dual-stream features through a multi-head self-attention mechanism (8 heads, key / query dimension of 64). The spatial features are calculated as the key, the temporal features as the value, a query is generated based on the speech energy spectrum, and the fusion weight matrix is ​​output, which is expressed as a formula: ; in, , representing the query vector, is generated by linear mapping from the speech energy spectrum and represents the dimension of the pronunciation feature that needs to be focused on; , representing the key vector (Key), which comes from the MFCC features of the spatial feature stream and encodes static articulation posture; , representing the value vector, which comes from the TCN output of the temporal feature stream and encodes dynamic temporal evolution; , representing the dimension of the key / query vector, used to scale the dot product to prevent gradient explosion.

[0043] Based on the above description, this application fuses static spatial features and dynamic temporal features through an attention weight matrix to obtain initial fused features. Then, the processing result of the 8-head self-attention mechanism on the static spatial features is used as the key, and the processing result of the 8-head self-attention mechanism on the dynamic temporal features is used as the value. Finally, the initial fused features are superimposed on the static spatial features and the dynamic temporal features respectively through residual connection, and the key and value are used as features for filtering to obtain fused features.

[0044] Cross-modal fusion enhances the semantic association between phonemes and lip shapes, ensuring the generalization of the character and achieving an identity retention rate of 92.4% in the Few-shot scenario (baseline 73.5%).

[0045] S103: Based on the biomechanical constraint algorithm, perform trajectory calculation on the fused features to generate key point motion trajectories that conform to biomechanical constraints.

[0046] In some embodiments of this application, after obtaining the fused features, trajectory calculations need to be performed on the fused features to generate keypoint motion trajectories that conform to biomechanical constraints. However, before that, it should be noted that this application proposes a trajectory optimization algorithm based on Gaussian process regression (GPR) to construct the covariance matrix of the second derivative of the motion of adjacent keypoints: ; In this study, lip movement data from 100 subjects were collected using Kinect V2, and the distribution of lip acceleration in natural dialogue scenarios was statistically analyzed to show that it follows a certain pattern. (95% confidence interval), determined by t-test (p<0.01). A reasonable threshold; , Indicates the timestamps (in seconds) of adjacent video frames, with a sampling interval of 0.04 seconds; Seconds represent a length scale parameter that controls the rate at which the correlation between the motions of adjacent keypoints decays. , representing the noise variance, is used to avoid matrix singularity.

[0047] The lip motion acceleration is constrained using the Lagrange multiplier method. Within the biomechanically feasible domain, eliminate non-physical mutations.

[0048] Solving for the optimal trajectory using the Lagrange multiplier method: ; The constraints are as follows: ; .

[0049] By using physical constraints for temporal smoothing, the spatiotemporal continuity of lip keypoint motion is modeled, and the acceleration changes in adjacent frames are constrained to conform to the smoothness of human motion.

[0050] Based on the above description, this application constructs the second derivative covariance matrix of adjacent keypoints with fused features using Gaussian process regression; optimizes the second derivative covariance matrix using the Lagrange algorithm according to preset jitter acceleration and coordinate accuracy weights to obtain the standard covariance matrix; finally, solves the standard covariance matrix by Cholesky decomposition to obtain the primary keypoint motion trajectory, and eliminates non-physical abrupt changes in the primary keypoint motion trajectory through physical constraints to generate a keypoint motion trajectory that conforms to biomechanical constraints.

[0051] By eliminating non-physical motion through biomechanical constraint algorithms, the smoothness of key point trajectory (SPL index) is improved by 2.3 times (from 0.45 to 1.03), the acceleration of the transition segment of plosive and friction sounds is reduced by 42%, and the motion trajectory conforms to the biomechanical laws of the human body.

[0052] S104: The motion trajectory of the key points is calibrated with the key point trajectory of the standardized target video through a joint loss function to generate the final facial key points; the joint loss function is related to the L1 loss function, the acceleration smoothing loss function and the physical constraint loss function.

[0053] In some embodiments of this application, the error between the motion trajectory of the keypoint and the trajectory of the standardized target video keypoint is calculated using the L1 loss function to calibrate the motion trajectory of the keypoint; the formula is: ; in For L1 loss, The coordinates of the facial key points predicted by the model (68 key points, each containing x and y coordinates) represent the motion state of facial regions such as lips and jaw generated in frame t. The coordinates of real-world labeled facial key points (extracted from the video using the OpenFace tool) serve as supervisory signals to guide the model in learning facial motion patterns that match the target character's identity. The L1 norm loss is calculated by summing the absolute differences between the predicted and actual values, which is used to directly constrain the generation accuracy of keypoint coordinates. The motion trajectories of key points outside the preset jitter acceleration range are calibrated using an acceleration smoothing loss function, so that the jitter acceleration of the key point motion trajectories falls within the preset jitter acceleration range; the formula is: ; The motion trajectories of key points outside the preset lip motion acceleration range are calibrated using a physical constraint loss function, so that the lip motion acceleration of the key point motion trajectories all fall within the preset lip motion acceleration range; the formula is: ; in The second derivative (i.e., acceleration) of the predicted keypoint trajectory is used to characterize the smoothness of the keypoint motion and avoid non-physical abrupt changes. It is calculated as follows: ; This represents the L2 norm regularization term, which calculates the sum of squares of accelerations to penalize high-frequency jitter and improve the temporal continuity of the generated trajectory. The total loss function formula is: ; in, , , The weight coefficients of the loss term are used to balance coordinate accuracy and motion smoothness. They are determined on the validation set through grid search to ensure that the gradient magnitudes of the three loss terms are similar.

[0054] The key points of the calibrated key point motion trajectory are determined as the final facial key points.

[0055] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S101 to S104 will be described sequentially, but this does not mean that steps S101 and S104 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1 The order in which steps S101 to S104 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S104 can be appropriately adjusted according to actual needs.

[0056] pass Figure 1This method constructs a spatial-temporal dual-stream feature decoupling architecture, reducing the lip-sync error (LSE-D) from the baseline of 4.1mm to 2.8mm, a reduction of 31.7%. This effectively solves the problem that traditional models cannot simultaneously process audio spatial and temporal features, improving the temporal alignment accuracy of lip movements and speech. By eliminating non-physical motion through a biomechanical constraint algorithm, the smoothness of keypoint trajectories (SPL index) is improved by 2.3 times (from 0.45 to 1.03), and the acceleration of plosive and fricative transition segments is reduced by 42%, with motion trajectories conforming to human biomechanical principles. Cross-modal fusion enhances the semantic association between phonemes and lip movements, achieving an identity retention rate of 92.4% in Few-shot scenarios (compared to 73.5% at baseline), significantly improving cross-role generalization ability. Attention mechanisms focus on key articulation regions, effectively capturing the deep semantic association between phonemes and facial muscle movements, and the expressive power of MFCC features overcomes the limitations of "hand-designed" approaches.

[0057] Figure 2 This is a spatiotemporal feature decoupling model framework diagram for a digital human-driven keypoint generation method provided in an embodiment of this application.

[0058] exist Figure 2 The paper presents the overall framework of the spatiotemporal feature decoupling model constructed in this application, including temporal feature flow and spatial feature flow.

[0059] Figure 3 A schematic diagram of a digital human-driven keypoint generation device provided in this application embodiment includes: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, A digital human-driven keypoint generation method is described, wherein the memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform any of the above-mentioned tasks.

[0060] Some embodiments of this application provide a non-volatile computer storage medium for generating digital human-driven key points, which stores computer-executable instructions capable of executing any of the above-described digital human-driven key point generation methods.

[0061] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0062] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0063] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0068] Memory may include non-persistent storage in computer-readable media, random access memory (RAM), and non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0069] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0070] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0071] The above are merely embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the technical principles of this application should fall within the protection scope of this application.

Claims

1. A method for generating key points driven by a digital human, characterized in that, The method includes: By using a pre-constructed spatial-temporal dual-stream feature decoupling model, static spatial features and dynamic temporal features of standardized target speech are extracted. According to the cross-modal dynamic attention fusion mechanism, the static spatial features and the dynamic temporal features are dynamically weighted and fused to obtain fused features; Based on the biomechanical constraint algorithm, trajectory calculation is performed on the fused features to generate key point motion trajectories that conform to biomechanical constraints. By using a joint loss function, the motion trajectory of the key points is calibrated with the key point trajectory of the standardized target video to generate the final facial key points; the joint loss function is related to the L1 loss function, the acceleration smoothing loss function, and the physical constraint loss function.

2. The method according to claim 1, characterized in that, The step of calibrating the motion trajectory of the key points with the key point trajectory of the standardized target video using a joint loss function to generate the final facial key points specifically includes: The error between the motion trajectory of the key point and the trajectory of the standardized target video key point is calculated using the L1 loss function to calibrate the motion trajectory of the key point. The motion trajectory of key points that is outside the preset jitter acceleration range is calibrated by using an acceleration smoothing loss function, so that the jitter acceleration of the motion trajectory of the key points falls within the preset jitter acceleration range. By using a physical constraint loss function, the motion trajectories of key points that are outside the preset lip motion acceleration range are calibrated so that the lip motion acceleration of the key point motion trajectories all fall within the preset lip motion acceleration range. The key points of the calibrated key point motion trajectory are determined as the final facial key points.

3. The method according to claim 1, characterized in that, The step of performing trajectory calculations on the fused features based on a biomechanical constraint algorithm to generate keypoint motion trajectories that conform to biomechanical constraints specifically includes: Based on the Gaussian process regression equation, construct the second derivative covariance matrix of adjacent key points of the fused features; Based on the preset jitter acceleration and coordinate accuracy weights, the second derivative covariance matrix is ​​optimized using the Lagrange algorithm to obtain the standard covariance matrix; The standard covariance matrix is ​​solved by Cholesky decomposition to obtain the primary keypoint motion trajectory. Non-physical abrupt changes in the primary keypoint motion trajectory are eliminated by physical constraints to generate keypoint motion trajectories that conform to biomechanical constraints.

4. The method according to claim 1, characterized in that, The method of dynamically weighting and fusing the static spatial features and the dynamic temporal features according to the cross-modal dynamic attention fusion mechanism to obtain fused features specifically includes: The initial fused features are obtained by fusing static spatial features and dynamic temporal features using an attention weight matrix; The processing results of the 8-head self-attention mechanism on static spatial features are used as keys, and the processing results of the 8-head self-attention mechanism on dynamic temporal features are used as values. The initial fused features are superimposed with static spatial features and dynamic temporal features through residual connections, and the fused features are obtained by filtering the features using keys and values ​​respectively.

5. The method according to claim 1, characterized in that, Before extracting the static spatial and dynamic temporal features of the target speech using a pre-constructed spatial-temporal dual-stream feature decoupling model, the method further includes: The target speech is determined based on the continuous speech sampling protocol and the preset speech sampling frequency; The target video is determined based on preset camera parameters and ambient brightness. The target speech and the target video are time-stamped according to the PTP protocol to achieve audio sampling synchronization.

6. The method according to claim 5, characterized in that, After aligning the target speech and the target video with timestamps according to the PTP protocol to achieve audio sampling synchronization, the method further includes: The RNNoise algorithm is used to improve the signal-to-noise ratio of the target speech to a preset signal-to-noise ratio. Feature extraction is performed on the enhanced target speech to obtain the third-level MFCC features of the target speech; The enhanced speech was annotated with phonemes using the Conformer ASR system to obtain the phoneme annotation results. The three-level MFCC features and the phoneme annotation results are integrated into a T×26-dimensional temporal feature matrix and output in NPZ format to obtain the standardized target speech; where T is the audio length.

7. The method according to claim 5, characterized in that, After aligning the target speech and the target video with timestamps according to the PTP protocol to achieve audio sampling synchronization, the method further includes: Face detection is performed on the target video using MediaPipe facial mesh to determine facial key points; The facial key points are subjected to feature standardization processing to obtain standardized features; The target video is enhanced in the temporal domain, spatial domain, and adversarial examples to obtain multi-dimensional enhanced data. The standardized features are integrated with the multi-dimensional enhanced data to generate a T×68×2 dimensional NPZ file, resulting in a standardized target video; where T is the number of video frames.

8. The method according to claim 1, characterized in that, Before extracting the static spatial and dynamic temporal features of the standardized target speech using a pre-constructed spatial-temporal dual-stream feature decoupling model, the method further includes: Construct a spatial-temporal dual-stream feature decoupling model; The spatial-temporal dual-stream feature decoupling model adopts the ResNet-50 architecture for its spatial feature stream, which includes 3 layers of 1D convolution and max pooling, and outputs a 256-dimensional MFCC feature vector. The spatial-temporal dual-stream feature decoupling model adopts a TCN architecture for its temporal feature stream, which includes four causal convolutional layers, each equipped with a gated activation unit (GLU).

9. A digital human-driven keypoint generation device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a digital human-driven keypoint generation method according to any one of claims 1-8.

10. A digital human-driven keypoint generation storage medium, storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of executing the digital human-driven keypoint generation method according to any one of claims 1-8.

Images