Method and device for synchronizing multiple materials of lip movements based on single training of real person

Abstract

The application provides a multi-material lip synchronization method and device based on single training of the same real person, which comprises the following steps: S1, constructing sample data containing facial features, speech features and face posture features; S2, training a posture parameter prediction sub-network, so that the face posture features output by the posture parameter prediction sub-network tend to the expected face posture features in the sample data; S3, performing coordinate transformation on the facial features, so as to map the parameters of the facial features to a standard coordinate system under a front view angle; S4, learning a dynamic mapping relationship between the fused features and the lip action through time sequence modeling, and obtaining an LSTM network capable of predicting the next frame of facial features; and S5, outputting the next frame of facial features of a digital person based on the trained LSTM network. The application simplifies the data acquisition process, improves the lip synchronization efficiency, reduces the cost, and improves the stability of the synchronization effect.

Description

A method and device for multi-material lip-syncing based on a single training session with the same real person Technical Field

[0001] This application belongs to the field of video processing technology, and specifically relates to a method and apparatus for lip-syncing of multiple materials based on a single training session with the same real person. Background Technology

[0002] Existing lip-sync algorithms often require separate training for each video clip when processing different video clips of the same person. This is because the actions, clothing, size, and position of the person in the video vary. This not only increases training time and cost, but also leads to inconsistencies between different trained models, resulting in unstable lip-sync performance and failing to meet the demands for efficient and stable lip-sync in practical applications. For example, in film and television post-production, virtual anchors, and animation production, different shots of the same actor or virtual character may have different actions, clothing, size, and position. Existing technologies significantly impact production efficiency. Furthermore, existing technologies have high requirements for data acquisition, requiring the collection of multiple sets of footage with different actions and clothing, increasing the complexity of pre-production preparation. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, this application provides a multi-material lip-syncing method and apparatus based on single-training of the same real person. By collecting a single video clip containing multiple facial angles and speaking actions of the same real person, along with corresponding speech data, core facial features and speech features are extracted. A general model is constructed and trained once, which can then be applied to lip-syncing inference of video clips featuring different actions, clothing, sizes, and positions of the person.

[0004] The first aspect of this application, a multi-material lip-syncing method based on a single training session with the same real person, mainly includes:

[0005] Step S1: Collect facial features of the same real person at different facial angles and the corresponding voice features to construct sample data containing facial features, voice features and facial pose features, wherein the facial pose features are used to characterize the facial angle.

[0006] Step S2: Input the sample data into the pose parameter prediction sub-network and train the pose parameter prediction sub-network so that the face pose features output by the pose parameter prediction sub-network tend to the expected face pose features in the sample data.

[0007] Step S3: Based on the facial pose features output by the pose parameter prediction subnetwork, perform coordinate transformation on the facial features to map the parameters of the facial features to the standard coordinate system under the frontal view.

[0008] Step S4: The mapped facial features and speech features are fused through an attention mechanism and input into the LSTM network. The dynamic mapping relationship between the fused features and lip movements is learned through temporal modeling to obtain an LSTM network that can predict facial features in the next frame.

[0009] Step S5: Obtain the facial features of the original video frame of the digital human that is consistent with the real person and the speech features of the next frame of the digital human that is to be broadcast by adapting to changes in lip shape. Output the facial features of the next frame of the digital human based on the trained LSTM network.

[0010] Preferably, in step S1, facial features and speech features are extracted within a specified number of frames. The facial features are represented by the positions of multiple facial feature points in a static state and their final displacement vectors within the specified number of frames. The speech features are represented by the Mel-frequency cepstral coefficients, fundamental frequency, speech rate, and pitch within the specified number of frames.

[0011] Preferably, step S1 further includes: expanding the sample data by scaling and translation data augmentation, with the scaling factor controlled between 0.8 and 1.2 times and the translation parameter controlled within 50 pixels.

[0012] Preferably, step S5 further includes:

[0013] Step S51: Scale the facial features of the digital human according to the scaling ratio of the digital human's facial size to the real human's facial size in the sample data.

[0014] Step S52: Based on the offset size of the digital human's facial center compared to the real human's facial center in the sample data, perform translation correction on the digital human's facial features;

[0015] Step S53: Input the corrected facial features and speech features into the LSTM network to obtain the predicted facial features for the next frame.

[0016] Step S54: Perform reverse scaling and reverse translation correction on the output facial features according to the scaling ratio and offset size to obtain the final facial features of the digital human in the next frame.

[0017] Preferably, step S5 further includes:

[0018] Step S6: Adapt and deform the entire mouth area of ​​the digital human according to the facial features of the next frame output;

[0019] Step S7: Based on the positional relationship between the facial features of the next output frame and the facial features of the original video frame, extract texture information from the original video frame using bilinear interpolation and map it to the deformed region. For texture stretching areas caused by deformation, a local texture synthesis algorithm is used to generate natural texture details.

[0020] Step S8: Based on the positional relationship between the facial features of the next frame and the facial features of the original video frame, apply the illumination parameters of each facial feature point in the facial features of the original video frame to the corresponding facial feature point in the next frame.

[0021] A second aspect of this application discloses a multi-material lip-syncing device based on a single training session with the same human subject, mainly comprising:

[0022] The sample construction module is used to collect facial features of the same real person at different facial angles and the corresponding voice features, and construct sample data containing facial features, voice features and facial pose features, wherein the facial pose features are used to characterize the facial angle.

[0023] The pose parameter prediction subnetwork training module is used to input the sample data into the pose parameter prediction subnetwork and train the pose parameter prediction subnetwork so that the face pose features output by the pose parameter prediction subnetwork tend to the expected face pose features in the sample data.

[0024] The facial feature transformation module is used to perform coordinate transformation on the facial features based on the facial pose features output by the pose parameter prediction subnetwork, so as to map the parameters of the facial features to a standard coordinate system under a frontal view.

[0025] The LSTM network training module is used to fuse the mapped facial features and speech features through an attention mechanism and input them into the LSTM network. Through temporal modeling, it learns the dynamic mapping relationship between the fused features and lip movements to obtain an LSTM network that can predict facial features in the next frame.

[0026] The facial feature prediction module is used to acquire the facial features of the original video frame of the digital human that is consistent with the real person, as well as the speech features of the next frame of the digital human that is to be broadcast by adapting lip movements. Based on the trained LSTM network, the module outputs the facial features of the digital human in the next frame.

[0027] Preferably, in the sample construction module, facial features and speech features are extracted within a specified number of frames. The facial features are represented by the positions of multiple facial feature points in a static state and their final displacement vectors within the multiple frames. The speech features are represented by the Mel-frequency cepstral coefficients, fundamental frequency, speech rate, and pitch within the multiple frames.

[0028] Preferably, the sample construction module includes a data augmentation unit, which is used to expand the sample data through data augmentation methods of scaling and translation, with the scaling factor controlled between 0.8 and 1.2 times and the translation parameter controlled within 50 pixels.

[0029] Preferably, the facial feature prediction module includes:

[0030] The scaling processing unit is used to scale the facial features of the digital human according to the scaling ratio of the digital human's facial size to the real human's facial size in the sample data.

[0031] The translation correction unit is used to perform translation correction on the facial features of the digital human based on the offset size of the digital human's facial center compared to the real human's facial center in the sample data;

[0032] The facial feature output unit is used to input the corrected facial features and speech features into the LSTM network to obtain the predicted facial features for the next frame.

[0033] The inverse transform unit is used to perform inverse scaling and inverse translation correction on the output facial features according to the scaling ratio and offset size to obtain the final facial features of the digital human in the next frame.

[0034] Preferably, the device further includes:

[0035] The lip shape deformation module is used to adapt and deform the entire lip shape area of ​​the digital human according to the facial features of the next frame output.

[0036] The texture processing module extracts texture information from the original video frame and maps it to the deformed region based on the positional relationship between the facial features of the next output frame and the facial features of the original video frame using bilinear interpolation. For texture stretching areas caused by deformation, a local texture synthesis algorithm is used to generate natural texture details.

[0037] The illumination matching module is used to apply the illumination parameters of each facial feature point in the original video frame to the corresponding facial feature point in the next frame based on the positional relationship between the facial features of the next frame and the facial features of the original video frame.

[0038] This application simplifies the data acquisition process, improves lip-sync efficiency, reduces costs, and enhances the stability of synchronization effects. Attached Figure Description

[0039] Figure 1 is a flowchart of an embodiment of the multi-material lip-syncing method based on a single training session with the same real person, as described in this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are only some, not all, of the embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0041] The first aspect of this application provides a multi-source lip-syncing method based on a single training session with the same real person, as shown in Figure 1, which mainly includes:

[0042] Step S1: Collect facial features of the same real person at different facial angles and the corresponding voice features to construct sample data containing facial features, voice features and facial pose features, wherein the facial pose features are used to characterize the facial angle.

[0043] Step S2: Input the sample data into the pose parameter prediction sub-network and train the pose parameter prediction sub-network so that the face pose features output by the pose parameter prediction sub-network tend to the expected face pose features in the sample data.

[0044] Step S3: Based on the facial pose features output by the pose parameter prediction subnetwork, perform coordinate transformation on the facial features to map the parameters of the facial features to the standard coordinate system under the frontal view.

[0045] Step S4: The mapped facial features and speech features are fused through an attention mechanism and input into the LSTM network. The dynamic mapping relationship between the fused features and lip movements is learned through temporal modeling to obtain an LSTM network that can predict facial features in the next frame.

[0046] Step S5: Obtain the facial features of the original video frame of the digital human that is consistent with the real person and the speech features of the next frame of the digital human that is to be broadcast by adapting to changes in lip shape. Output the facial features of the next frame of the digital human based on the trained LSTM network.

[0047] This application only requires a single video clip containing lip movements from multiple angles of a person's face while speaking for training once. This training can then be applied to lip-sync inference for video clips of the same person with different actions, clothing, size, and position, thereby simplifying the data collection process and improving efficiency.

[0048] Specifically, this application first acquires video footage of real people in step S1. This video footage typically refers to various angles of a person's face when they speak (such as frontal, 30° side angle, 45° side angle, upward angle, downward angle, etc.) and various speaking actions (such as normal speaking speed, fast speaking, slow speaking, soft speaking, stressed speaking, etc.). During filming, the video resolution is no less than 1920×1080, the frame rate is 25-30fps, and the lighting intensity is maintained at 300-800 lux to avoid direct sunlight or backlighting that could blur facial features. The speech sampling rate is 16kHz, mono, and the signal-to-noise ratio is no less than 30dB to ensure clear speech without noise.

[0049] Then, the core facial feature points of the real person were extracted from the collected video footage at different angles, and voice features were extracted from the voice data.

[0050] In some optional implementations, in step S1, facial features and speech features are extracted within a specified number of frames. The facial features are represented by the positions of multiple facial feature points in a static state and their final displacement vectors within the multiple frames. The speech features are represented by the Mel-frequency cepstral coefficients, fundamental frequency, speech rate, and pitch within the multiple frames.

[0051] Specifically, for facial features, the three-dimensional pose parameters of the face in each frame of the image can be obtained first through a 3D facial pose estimation algorithm, namely the facial pose features, including pitch angle, yaw angle and roll angle. Then, the Dlib feature point detection model is dynamically adjusted according to the pose parameters to achieve better feature point extraction. The extracted feature points mainly include: lip contour (10 feature points for the upper lip and 10 feature points for the lower lip), the connection between the upper and lower jaws (2 feature points), the area on both sides of the nose that is linked with the lips (4 feature points), and the muscle group extending from the corners of the mouth (4 feature points), for a total of 30 core feature points. Each facial feature point records its coordinates (x, y) in the image coordinate system and its displacement vector that changes over time.

[0052] For speech features, the librosa library in Python is used to extract speech features, including 39-dimensional me-frequency cepstral coefficients, fundamental frequency (calculated by the YIN algorithm, ranging from 50 to 500 Hz), speech rate (statistically calculated by speech frame interval, in syllables / second), and pitch (calculated by the fundamental frequency change rate, in Hz / second). All speech features are then normalized and mapped to the interval [-1, 1].

[0053] In the above embodiments, sample data is acquired through a sliding window, and data is extracted once every 5ms, with each data set containing multiple frames with a duration of 10ms.

[0054] Thus, in step S1, a total of 30 facial core feature points (x, y) coordinates and displacement vectors (30×3=90 dimensions) and speech features (42 dimensions) are formed, totaling 132 dimensions of input features. The three-dimensional facial pose features are used as the expected output.

[0055] In some optional implementations, step S1 further includes: augmenting the sample data by scaling and translation, with the scaling factor controlled between 0.8 and 1.2 times and the translation parameter controlled within 50 pixels.

[0056] This embodiment enables the subsequent training model to improve its generalization ability during the learning process, ensuring that it can directly adapt to the scale and positional differences of new materials after a single training session.

[0057] Next, in step S2, the pose parameter prediction subnetwork is trained to output facial pose features. Then, in step S3, the facial features are subjected to coordinate transformation. The transformed facial features are used to train the LSTM network in step S4.

[0058] It is understandable that the pose parameter prediction subnetwork and the LSTM network can be trained separately or combined. When trained together, the pose parameter prediction subnetwork is part of the LSTM network.

[0059] Taking merged training as an example, in addition to the input layer that receives 132-dimensional input features, the LSTM network also includes:

[0060] Angle adaptation layer: It consists of two parallel sub-networks. The first is the pose parameter prediction sub-network, which adopts a 3-layer fully connected network with 128, 64 and 3 hidden units respectively. This sub-network can output 3D face pose features. The second is the feature transformation sub-network, which generates an affine transformation matrix based on the predicted face pose features and performs coordinate transformation on the input face feature points, mapping face feature points at different angles to the standard coordinate system under the frontal view to eliminate angle differences.

[0061] Feature fusion layer: An attention mechanism is used to fuse standardized facial and speech features. First, a fully connected layer maps facial feature points and speech features to a 128-dimensional feature space. Then, the attention weight matrix of the two is calculated, and feature weight fusion is achieved through matrix multiplication, highlighting the correlation weight between lip features and speech fundamental frequency and speech rate. The fused feature dimension is 128.

[0062] Mapping layer: It consists of a backbone of a 3-layer bidirectional LSTM network with 256 hidden units in each layer. It uses dropout (a regularization technique) to prevent the model from overfitting and learns the dynamic mapping relationship between fused features and lip movements through temporal modeling.

[0063] Output layer: A fully connected layer is used to output the predicted coordinates of 30 facial feature points for the next frame, which are used as lip movement parameters.

[0064] This application divides the sample data into a training set and a validation set in an 8:2 ratio. The mean squared error between the predicted coordinates and the true coordinates is calculated using the MSE loss function. The optimizer is Adam, with an initial learning rate of 0.001. When the validation set loss does not decrease for 5 consecutive rounds, the learning rate is reduced to 1 / 10 of its original value. The total number of training rounds is 80. Training is stopped when the validation set loss is lower than 0.005, and the model parameters are saved.

[0065] Once the trained LSTM network is in place, the lip movements of the digital human can be synchronized in step S5. This requires acquiring the original video frames containing the digital human to obtain the facial features of the digital human within those frames. Simultaneously, audio data needs to be acquired to determine the speech features of the next frame. Then, based on the LSTM network, the facial features of the next frame are determined according to the speech features of that frame.

[0066] Because the original video frame is inconsistent with the video frame used for LSTM network training, it needs to be preprocessed. This includes resolution unification (scaling to 1920×1080), frame rate normalization (converting to 30fps), Gaussian filtering to remove image noise, denoising the audio (using spectral subtraction), and unifying the sampling rate (converting to 16kHz). Simultaneously, an object detection algorithm (such as YOLOv5) is used to locate the facial region of the person in the original video frame, obtaining the face's width, height, and center coordinates.

[0067] In some alternative implementations, step S5 further includes:

[0068] Step S51: Scale the facial features of the digital human according to the scaling ratio of the digital human's facial size to the real human's facial size in the sample data.

[0069] Step S52: Based on the offset size of the digital human's facial center compared to the real human's facial center in the sample data, perform translation correction on the digital human's facial features;

[0070] Step S53: Input the corrected facial features and speech features into the LSTM network to obtain the predicted facial features for the next frame.

[0071] Step S54: Perform reverse scaling and reverse translation correction on the output facial features according to the scaling ratio and offset size to obtain the final facial features of the digital human in the next frame.

[0072] In step S51, the scaling factor for the facial dimensions of the digital human in the original video frame and the real person in the training sample data is calculated:

[0073] scale=min(W / W0,H / H0);

[0074] Where W0 is the face width of the real person in the training sample data, H0 is the face height of the real person in the training sample data, W is the face width of the digital person in the original video frame, and H is the face height of the digital person in the original video frame.

[0075] The coordinates (x, y) of each facial feature point extracted from the original video frame are inversely normalized according to a scaling factor, using the following formula:

[0076] xnorm=x / scale,ynorm=y / scale;

[0077] Where (xnorm, ynorm) are the scaled coordinates of each facial feature point. This processing method enables the facial feature point coordinates in the original video frame to be mapped to the standard scale space of the training samples.

[0078] Next, in step S52, person position matching is performed. First, the center coordinates (Cx, Cy) of the digital face in the original video frame are calculated, and then compared with the standard center coordinates (Cx0, Cy0) of the real face in the training samples to determine the offset (dx, dy).

[0079] dx = Cx - Cx0, dy = Cy - Cy0.

[0080] This allows for the translation and correction of the coordinates of facial feature points in the original video frame, using the following formula:

[0081] xtrans=xnorm-dx, ytrans=ynorm-dy;

[0082] Where (xtrans, ytrans) are the coordinates of each facial feature point after translation.

[0083] In step S53, the LSTM network outputs the facial features of the next frame, and finally in step S54, an inverse transformation is performed to match a digital person with actual size and position.

[0084] In some alternative implementations, step S5 is further followed by:

[0085] Step S6: Adapt and deform the entire mouth area of ​​the digital human according to the facial features of the next frame output;

[0086] Step S7: Based on the positional relationship between the facial features of the next output frame and the facial features of the original video frame, extract texture information from the original video frame using bilinear interpolation and map it to the deformed region. For texture stretching areas caused by deformation, a local texture synthesis algorithm is used to generate natural texture details.

[0087] Step S8: Based on the positional relationship between the facial features of the next frame and the facial features of the original video frame, apply the illumination parameters of each facial feature point in the facial features of the original video frame to the corresponding facial feature point in the next frame.

[0088] This embodiment mainly describes the post-rendering process.

[0089] In step S6, based on the facial features output by the model, an improved Moving Least Squares (MLS) method is used to deform the mouth region. First, a dense triangular mesh is constructed between the inner and outer contours of the lips, for example, one vertex every 2-3 pixels, resulting in approximately 200-300 triangles. Then, the target position of each mesh vertex is calculated based on the coordinates of the output facial feature points. The deformation field is solved using the MLS algorithm to ensure that the deformed mesh maintains its topological structure and has a natural transition. The deformation weights are set as follows: lip contour feature points have a weight of 1.0, corner of the mouth muscle feature points have a weight of 0.8, and other facial feature points have weights of 0.3-0.5 (decreasing with increasing distance from the lip region).

[0090] In step S7, a manifold-based texture transfer algorithm is used to establish a correspondence between the original video frame and the deformed lip region. First, the lip region is divided into multiple triangular patches using Delaunay triangulation. The affine transformation matrix of each patch in the original and target frames is calculated. Then, texture information is extracted from the original video frame using bilinear interpolation and mapped to the deformed region. For texture stretching areas caused by deformation, a local texture synthesis algorithm (such as PatchMatch) is used to generate natural texture details.

[0091] Step S8 is used for lighting matching. During the rendering process, the local lighting characteristics (including brightness, contrast, and color distribution) of the original video lip-sync area are analyzed, and the lighting difference between the original area and the target deformed area is calculated. Using techniques such as gamma correction, histogram matching, and color balancing, the lighting parameters of the deformed area are adjusted to match the lighting conditions of the surrounding facial areas. Specifically, the V channel (brightness) and S channel (saturation) parameters from the HSV color space of the original area are extracted and applied to the deformed area to maintain the overall facial tone.

[0092] Alternative implementation methods also include video post-processing techniques such as anti-aliasing, rendering pipeline optimization, and boundary constraints.

[0093] For anti-aliasing, this application employs Multi-Sample Anti-Aliasing (MSAA) to handle deformed edges. During rendering, 4-8 sample points are sampled for each pixel, the color value of each sample point is calculated, and then a weighted average is used to obtain the final pixel value. For deformed edge regions, the Fast Approximate Anti-Aliasing (FXAA) algorithm is additionally applied. By detecting color abrupt changes at the edges and blurring them, the jagged edges are eliminated, resulting in a smoother and more natural transition at the mouth shape edges.

[0094] For rendering pipeline optimization, the rendering process is divided into multiple stages for parallel processing, including feature point processing (completed by the CPU), deformation calculation (completed by the GPU), texture sampling (completed by the GPU), lighting matching (completed by the GPU), and final compositing (completed by the GPU). Texture caching is used to store intermediate results, avoiding redundant calculations. For minor changes between consecutive frames, inter-frame prediction is used to calculate only the changed areas, increasing rendering speed to over 25fps to meet the needs of real-time applications.

[0095] For boundary constraints, a "boundary check mechanism" is added to the rendering logic. When the predicted feature points are mapped to the original coordinates of the new video, if they exceed the facial area (e.g., the lip feature points exceed the facial boundary due to excessive scaling), a local adjustment is triggered: a safe area is calculated based on the facial bounding box (the boundary shrinks inward by 5%), and the facial feature points that exceed the area are clamped; the facial feature point weights are dynamically adjusted in combination with the pose of the digital human in the original video frame (e.g., the face width is narrower when the face is in profile) to avoid lip distortion caused by size adaptation.

[0096] In summary, this application only requires a single video clip of the same real person (containing multiple facial angles and speaking actions) for training once, which can then be applied to lip-sync inference for video clips of the same person with different actions, clothing, sizes, and positions. This greatly simplifies the data collection process, reduces preliminary preparation work, and lowers time and costs. Because the same model is used for inference, differences between different models are avoided, improving the stability of the lip-sync effect. This application extracts core facial feature points and speech features from multiple angles and combines them with a deep learning network capable of adapting to angles, sizes, and positions to construct a general model. This model can accurately capture the correspondence between the real person's lip movements and speech under different angles, sizes, and positions, improving the accuracy and adaptability of lip-sync. This application employs a unique rendering logic design, including improved lip deformation algorithms, texture processing, lighting matching, anti-aliasing technology, and boundary constraint mechanisms, resulting in natural and smooth rendered lip movements with high integration with the original video and better visual effects.

[0097] The second aspect of this application provides a multi-material lip-syncing device based on a single training session with the same real person, corresponding to the above method, mainly comprising:

[0098] The sample construction module is used to collect facial features of the same real person at different facial angles and the corresponding voice features, and construct sample data containing facial features, voice features and facial pose features, wherein the facial pose features are used to characterize the facial angle.

[0099] The pose parameter prediction subnetwork training module is used to input the sample data into the pose parameter prediction subnetwork and train the pose parameter prediction subnetwork so that the face pose features output by the pose parameter prediction subnetwork tend to the expected face pose features in the sample data.

[0100] The facial feature transformation module is used to perform coordinate transformation on the facial features based on the facial pose features output by the pose parameter prediction subnetwork, so as to map the parameters of the facial features to a standard coordinate system under a frontal view.

[0101] The LSTM network training module is used to fuse the mapped facial features and speech features through an attention mechanism and input them into the LSTM network. Through temporal modeling, it learns the dynamic mapping relationship between the fused features and lip movements to obtain an LSTM network that can predict facial features in the next frame.

[0102] The facial feature prediction module is used to acquire the facial features of the original video frame of the digital human that is consistent with the real person, as well as the speech features of the next frame of the digital human that is to be broadcast by adapting lip movements. Based on the trained LSTM network, the module outputs the facial features of the digital human in the next frame.

[0103] In some optional implementations, in the sample construction module, facial features and speech features are extracted within a specified number of frames. The facial features are represented by the positions of multiple facial feature points in a static state and their final displacement vectors within the multiple frames. The speech features are represented by Mel-frequency cepstral coefficients, fundamental frequency, speech rate, and pitch within the multiple frames.

[0104] In some optional implementations, the sample construction module includes a data augmentation unit for expanding the sample data through scaling and translation data augmentation, with the scaling factor controlled between 0.8 and 1.2 times and the translation parameter controlled within 50 pixels.

[0105] In some alternative implementations, the facial feature prediction module includes:

[0106] The scaling processing unit is used to scale the facial features of the digital human according to the scaling ratio of the digital human's facial size to the real human's facial size in the sample data.

[0107] The translation correction unit is used to perform translation correction on the facial features of the digital human based on the offset size of the digital human's facial center compared to the real human's facial center in the sample data;

[0108] The facial feature output unit is used to input the corrected facial features and speech features into the LSTM network to obtain the predicted facial features for the next frame.

[0109] The inverse transform unit is used to perform inverse scaling and inverse translation correction on the output facial features according to the scaling ratio and offset size to obtain the final facial features of the digital human in the next frame.

[0110] In some alternative embodiments, the apparatus further includes:

[0111] The lip shape deformation module is used to adapt and deform the entire lip shape area of ​​the digital human according to the facial features of the next frame output.

[0112] The texture processing module extracts texture information from the original video frame and maps it to the deformed region based on the positional relationship between the facial features of the next output frame and the facial features of the original video frame using bilinear interpolation. For texture stretching areas caused by deformation, a local texture synthesis algorithm is used to generate natural texture details.

[0113] The illumination matching module is used to apply the illumination parameters of each facial feature point in the original video frame to the corresponding facial feature point in the next frame based on the positional relationship between the facial features of the next frame and the facial features of the original video frame.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for lip-syncing using multiple materials based on a single training session with the same real person, characterized in that, include: Step S1: Collect facial features of the same person at different facial angles and corresponding speech features to construct sample data containing facial features, speech features, and facial pose features. The facial pose features are used to represent facial angles. Step S2: Input the sample data into a pose parameter prediction subnetwork to train the subnetwork so that the facial pose features output by the subnetwork tend to the expected facial pose features in the sample data. Step S3: Perform coordinate transformation on the facial features based on the facial pose features output by the pose parameter prediction subnetwork to map the parameters of the facial features to a standard coordinate system under a frontal view. Step S4: Fuse the mapped facial features and speech features through an attention mechanism and input them into an LSTM network. Learn the fused features and lip movements through temporal modeling. The dynamic mapping relationship of actions is obtained to obtain an LSTM network capable of predicting facial features in the next frame; Step S5: Obtain the facial features of the original video frame of the digital human consistent with the real person and the speech features of the next frame of the digital human to be broadcast by adapting lip movements. Output the facial features of the digital human in the next frame based on the trained LSTM network; In Step S4, the mapped facial features and speech features are fused through an attention mechanism, including first mapping the facial feature points and speech features to a 128-dimensional feature space through a fully connected layer, then calculating the attention weight matrix of the two, and achieving feature weighted fusion through matrix multiplication; When learning the dynamic mapping relationship between fused features and lip movements through temporal modeling, the mapping layer is composed of the backbone of a 3-layer bidirectional LSTM network with 256 hidden units in each layer, and dropout technology is used to prevent model overfitting.

2. The multi-material lip-syncing method based on a single training session with the same real person, as described in claim 1, is characterized in that... In step S1, facial features and speech features are extracted within a specified number of frames. The facial features are represented by the positions of multiple facial feature points in a static state and their final displacement vectors within the specified number of frames. The speech features are represented by the Mel-frequency cepstral coefficients, fundamental frequency, speech rate, and pitch within the specified number of frames.

3. The multi-material lip-syncing method based on a single training session with the same real person, as described in claim 1, is characterized in that... Step S1 further includes: expanding the sample data through data augmentation methods of scaling and translation, with the scaling factor controlled between 0.8 and 1.2 times and the translation parameter controlled within 50 pixels.

4. The multi-material lip-syncing method based on a single training session with the same real person, as described in claim 1, is characterized in that... Step S5 further includes: Step S51, scaling the digital human's facial features according to the scaling ratio of the digital human's facial size to the real human's facial size in the sample data; Step S52, performing translation correction on the digital human's facial features according to the offset size of the digital human's facial center compared to the real human's facial center in the sample data; Step S53, inputting the corrected facial features and speech features into the LSTM network to obtain the predicted facial features for the next frame; Step S54, performing reverse scaling and reverse translation correction on the output facial features according to the scaling ratio and offset size to obtain the final facial features for the next frame of the digital human.

5. The multi-material lip-syncing method based on a single training session with the same real person, as described in claim 1, is characterized in that... Step S5 is followed by: Step S6, adapting and deforming the entire mouth area of ​​the digital human according to the facial features of the next output frame; Step S7, extracting texture information from the original video frame and mapping it to the deformed area by bilinear interpolation according to the positional relationship between the facial features of the next output frame and the facial features of the original video frame; for texture stretching areas caused by deformation, a local texture synthesis algorithm is used to generate natural texture details; Step S8, applying the lighting parameters of each facial feature point in the facial features of the original video frame to the corresponding facial feature points in the next frame according to the positional relationship between the facial features of the next output frame and the facial features of the original video frame.

6. A multi-material lip-syncing device based on a single training session with the same real person, characterized in that, include: The sample construction module collects facial features of the same person from different facial angles and corresponding speech features to construct sample data containing facial features, speech features, and facial pose features, where the facial pose features represent facial angles. The pose parameter prediction sub-network training module inputs the sample data into the pose parameter prediction sub-network to train it, ensuring that the facial pose features output by the sub-network approximate the expected facial pose features in the sample data. The facial feature transformation module performs coordinate transformation on the facial features based on the facial pose features output by the pose parameter prediction sub-network, mapping the parameters of the facial features to a standard coordinate system under a frontal view. The LSTM network training module fuses the mapped facial features and speech features through an attention mechanism and outputs the data. The system is fed into an LSTM network, where temporal modeling is used to learn the dynamic mapping relationship between fused features and lip movements, resulting in an LSTM network capable of predicting facial features in the next frame. The facial feature prediction module acquires facial features from the original video frames of the digital human, consistent with the real person, and the speech features of the next frame from which the digital human intends to adapt lip movements for speech delivery. Based on the trained LSTM network, it outputs the facial features of the digital human in the next frame. The LSTM network training module includes a feature fusion layer and a mapping layer. The feature fusion layer first maps facial feature points and speech features to a 128-dimensional feature space through a fully connected layer, then calculates the attention weight matrix of both, achieving weighted feature fusion through matrix multiplication. The mapping layer consists of a 3-layer bidirectional LSTM network backbone, with 256 hidden units per layer, and uses dropout technology to prevent overfitting.

7. The multi-material lip-syncing device based on a single training session with the same real person according to claim 6, characterized in that, In the sample construction module, facial features and speech features are extracted within a specified number of frames. The facial features are represented by the positions of multiple facial feature points in a static state and their final displacement vectors within the multiple frames. The speech features are represented by the Mel-frequency cepstral coefficients, fundamental frequency, speech rate, and pitch within the multiple frames.

8. The multi-material lip-syncing device based on a single training session with the same real person according to claim 6, characterized in that, The sample construction module includes a data augmentation unit, which is used to expand the sample data through scaling and translation data augmentation methods. The scaling factor is controlled between 0.8 and 1.2 times, and the translation parameter is controlled within 50 pixels.

9. The multi-material lip-syncing device based on a single training session with the same real person according to claim 6, characterized in that, The facial feature prediction module includes: a scaling processing unit, used to scale the facial features of the digital human according to the scaling ratio of the digital human's facial size to the real human's facial size in the sample data; a translation correction unit, used to perform translation correction on the digital human's facial features according to the offset size of the digital human's facial center compared to the real human's facial center in the sample data; a facial feature output unit, used to input the corrected facial features and speech features into an LSTM network to obtain the predicted facial features for the next frame; and an inverse transform unit, used to perform inverse scaling and inverse translation correction on the output facial features according to the scaling ratio and offset size to obtain the final facial features of the digital human for the next frame.

10. The multi-material lip-syncing device based on a single training session with the same real person according to claim 6, characterized in that, The device further includes: a lip-shape deformation module, used to adapt and deform the entire lip-shape region of the digital human according to the facial features of the next output frame; a texture processing module, used to extract texture information from the original video frame and map it to the deformed region by bilinear interpolation according to the positional relationship between the facial features of the next output frame and the facial features of the original video frame; for texture stretching areas caused by deformation, a local texture synthesis algorithm is used to generate natural texture details; and an illumination matching module, used to apply the illumination parameters of each facial feature point in the facial features of the original video frame to the corresponding facial feature point in the next frame according to the positional relationship between the facial features of the next output frame and the facial features of the original video frame.

Images