Personalized digital human generation method based on single video

Abstract

The invention discloses a personalized digital human generation method based on a single video, and relates to the field of virtual digital human modeling and driving, and the method aims at the video and voice data of a target person, through introducing a multi-modal alignment constrained voice driving synchronization mechanism and combining semantic understanding and an emotion label expression generation model, a personalized digital human model is generated. High synchronization, nature and vividness of digital human facial expressions and voice contents are realized. According to the method, a self-supervised style consistency constraint is added in model training to ensure that the style of a generated character image is stable, and a 3D semantic mask is added in an image fusion stage to improve the fusion precision and realistic effect of a synthetic facial expression and a reference face. According to the method, the digital human can be rapidly cloned and driven to synthesize the expression through the voice only through a single video sample, the generation process is efficient, and the obtained digital human video has excellent sense of reality and interactivity.

Description

Technical Field

[0001] The present invention relates to the field of virtual digital human modeling and driving, and particularly to a personalized digital human generation method based on a single video. Background Art

[0002] In the fields of computer vision and artificial intelligence, personalized digital human generation technology is one of the core research directions for constructing virtual characters and enhancing the human-computer interaction experience. The current mainstream technical paths are mainly divided into two categories: one is the traditional solution based on 3D modeling and blend shape deformation, and the other is the generation solution based on 2D key point driving or end-to-end neural networks. The traditional 3D modeling method can achieve precise facial expression control by manually constructing a high-precision 3D face model and predefining BlendShape parameters. However, this method has significant defects: firstly, the modeling process relies on a large amount of manual intervention, and it is necessary to design expression bases separately for each target person, resulting in a long modeling cycle and high costs; secondly, the generalization ability is poor, and it is impossible to quickly adapt to new characters or complex expression combinations; thirdly, the real-time rendering of 3D models requires extremely high hardware computing power and is difficult to run smoothly on mobile devices or low-power devices. Although the end-to-end 2D generation method has improved the real-time performance by simplifying the modeling process in recent years, its limitations are also obvious: on the one hand, it is difficult for the direct mapping based on 2D key points or neural networks to fully retain the unique facial features of the target person, and it is easy to produce the "average face" effect, resulting in the loss of personalized details; on the other hand, such methods usually rely on a large amount of training data to cover diverse facial expression patterns. When there is only a single video or a small number of samples, it is difficult for the model to generate high-fidelity results. In addition, the "black box" characteristic of the end-to-end system makes the controllability and interpretability of expression parameters insufficient, and developers cannot accurately adjust expression details or diagnose generation anomalies. Therefore, based on the above problems, the present invention proposes a personalized digital human generation method based on a single video. Summary of the Invention

[0003] Technical Objectives In order to solve the above problems, the objective of the present invention is to provide a personalized digital human generation method based on a single video, aiming to solve the problem of limited digital human applications caused by high modeling costs, low data efficiency, and insufficient real-time performance, and to realize the rapid construction of a high-fidelity digital image of the target person from a single video and the real-time driving of its facial expressions and lip movements in synchronization. While reducing the dependence on 3D modeling and large datasets, it improves the generation efficiency and cross-platform adaptation ability through modular design and lightweight deployment strategies.

[0004] Technical Solutions To achieve the above object, the present invention provides a method for generating a personalized digital human based on a single video. In this solution, a video of a target person is acquired in advance and a digital human face model of the target person is constructed, and a self-supervised style consistency constraint is introduced in model training to ensure stable generation style; the acoustic features of the speech input of the target person are extracted, and semantic understanding and emotion recognition are performed on the speech content; based on the acoustic features and semantic and emotion information, a corresponding target face image is generated under multi-modal alignment constraints; finally, the target face image is fused onto the benchmark face of the target person through a 3D semantic mask, and a personalized digital human video is output.

[0005] In a first aspect, the present invention provides a method for generating a personalized digital human based on a single video, including: Acquiring a video or several images of a target person, and constructing a digital human face model of the target person, and introducing a self-supervised style consistency loss during model training to ensure stable and consistent generation style; Acquiring the speech input of the target person and extracting acoustic feature parameters; Performing semantic understanding and emotion recognition on the speech input to obtain corresponding semantic information and emotion labels; Under multi-modal alignment constraints, mapping the acoustic feature parameters and the semantic information and emotion labels to a corresponding sequence of facial expression parameters; wherein a dynamic semantic feature control mechanism is introduced for the semantic stress and emotion intensity of the speech content to adjust the amplitude and details of the generated expression; According to the sequence of facial expression parameters, generating a face image of the target person through the digital human face model; the generation process includes a differentiable rendering layer for end-to-end expression synthesis of key facial regions, and combining style transfer constraints to ensure that the generated local expression images are consistent with the overall appearance style of the target person; Using a 3D semantic mask to fuse the face image of the target person onto a pre-stored benchmark face image of the target person, completing the fusion and superposition of the local expression region and the benchmark face at the pixel level, and outputting a personalized digital human video synchronized with the speech input content.

[0006] Further, the speech input is preprocessed and divided into continuous short-time frames, and Mel spectrogram coefficients or MFCC coefficient acoustic features are extracted. The acoustic features are input into an audio-driven model of a deep neural network to regress and output a sequence of facial expression parameters; the audio-driven model includes a long short-term memory network layer for capturing the temporal dependence of the speech signal, and further includes a speaker adaptation mechanism for fine-tuning some parameters of the model under the condition of only providing a small amount of speech-expression correspondence data of the target speaker, so as to adjust the expression parameter mapping to fit the pronunciation style and expression habit of the speaker.

[0007] Furthermore, the audio-driven model further includes an adjustable mapping unit for different speakers' speech rates and tone rhythms. The mapping unit dynamically adjusts the output rate and smoothness of the expression parameter sequence according to the phoneme duration, speech rate, and intonation fluctuations of the speech signal; by introducing additional features or modules reflecting the speech rhythm, it ensures a rapid response and stable transition of mouth movements at a fast speech rate, and maintains smooth and natural expression changes at a slow speech rate.

[0008] Furthermore, the audio-driven modeling process can also adopt a Transformer network, a temporal convolutional network, or other equivalent alternative structures.

[0009] Furthermore, the process of using semantic information and emotion labels to drive expression generation includes a context-semantic linkage mechanism for dynamically adjusting additional facial expression features such as eyebrows and eye expressions according to the semantic focus and emotion category in the speech content.

[0010] Furthermore, the process of generating a target human face expression image through a digital human face model includes using a differentiable rendering layer to perform expression synthesis rendering on local facial regions; the differentiable rendering layer receives the facial expression parameter sequence and combines it with the target person's face model to generate corresponding target person expression image frames.

[0011] Furthermore, the use of a 3D semantic mask for image fusion specifically includes: Step 1: Based on the three-dimensional human face model pre-constructed for the target person, extract the three-dimensional semantic masks of the mouth and periorbital regions; Step 2: Dynamically adjust the geometry and spatial position of the mask according to the expression parameters of the current frame to make it adapt to the opening and closing state of the target mouth shape; Step 3: Pixel-level blend the generated target human face local expression image with the pre-stored reference human face image within the fusion region defined by the three-dimensional semantic mask, and verify the alignment consistency of the orientation and scale of the fusion region and the reference human face through a spatial transformation matrix.

[0012] Furthermore, the method supports cross-platform deployment, including: Implement real-time inference on the mobile side through model compression and quantization; Support high-concurrency requests on the cloud side through distributed computing; Accelerate image synthesis on the web side through WebAssembly or WebGL.

[0013] In a second aspect, the present invention further provides a personalized digital human generation system based on a single video. The system is based on the method described in the first aspect above and includes: A video modeling module, which is used to construct a digital human face model of the target person based on a video or several images of the target person, and introduce a self-supervised style consistency loss during the model training process to ensure that the generated style is stable and consistent; A voice feature extraction module, which is used to extract the acoustic feature parameters of the voice input of the target person; A semantic and sentiment analysis module, which is used to perform semantic understanding and sentiment recognition on the voice input, and obtain the corresponding semantic information and sentiment labels; An expression generation module, which is used to map the acoustic feature parameters and the semantic information and sentiment labels into corresponding facial expression parameters under the constraint of multimodal alignment, and generate a face image of the target person according to the facial expression parameters; An image fusion module, which is used to fuse the face image of the target person into a pre-stored reference face image of the target person by using a 3D semantic mask, and output a personalized digital human video.

[0014] Furthermore, the voice feature extraction module further supports multimodal input, including using the audio generated by text-to-speech as a driving signal to realize the end-to-end mapping from text to digital human expressions.

[0015] Furthermore, it also includes an expression parameter interpretable adjustment module, which constrains and refines the facial expression parameter sequence; when the voice input is interrupted, the adjustment module maintains the current expression parameter state or makes the expression parameters smoothly transition to the neutral value; when any expression parameter exceeds the preset reasonable range, the adjustment module adjusts it within the threshold by means of clipping limitation or interpolation correction; the module also provides several interpretable control indicators for manual or algorithmic real-time adjustment of the expression intensity and style.

[0016] In a third aspect, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is run by a processor, it executes the aforementioned method for generating a personalized digital human based on a single video.

[0017] Through the coordination of three major modules: appearance modeling, audio-driven, and image synthesis, the present invention constructs a digital human driving framework with low data dependence and high real-time performance. The video modeling module uses principal component analysis to extract the geometric and texture features of the target person from a single video, constructs a low-dimensional expression basis space, and replaces traditional 3D modeling; the expression generation module uses a long short-term memory network to establish a temporal mapping from Mel spectrogram to expression parameters to achieve interpretable lip movement control; the image fusion module combines non-rigid deformation and generative adversarial network to generate high-fidelity continuous frames through local geometric adaptation and texture refinement. The present invention only needs a single video of the target person to clone and generate a digital human image, reducing data requirements and costs; by introducing semantic understanding and emotion-driven mechanisms, the facial expressions of the generated digital human can match the speech content and the speaker's emotions, making it more vivid and realistic; adopting multi-modal alignment constraints improves the synchronization accuracy of speech-driven lip movements and audio; using self-supervised style consistency loss ensures the style stability and consistency of the generated digital human in different frame sequences, avoiding image flickering and distortion; and combining 3D semantic mask fusion improves the geometric matching degree and naturalness of face synthesis, thus significantly enhancing the realism and interactive experience of the digital human video as a whole. This solution supports real-time operation across platforms through a modular architecture and a lightweight deployment strategy, significantly reducing the computing resource requirements, solving the problems of high data dependence, the contradiction between fidelity and efficiency, and weak platform adaptability, and providing an efficient and robust solution for virtual interaction scenarios.

[0018] Beneficial Effects By implementing the personalized digital human generation method based on a single video provided by the present invention, the following technical effects are achieved: (1) The present application extracts the expression change pattern of the target person from a single video through principal component analysis, constructs a low-dimensional orthogonal basis space representation, and replaces traditional 3D modeling or end-to-end black box models. This technology effectively reduces data requirements and computational complexity, while retaining personalized facial details, solves the dependence of traditional methods on multi-view data or large-scale annotation sets, and significantly improves modeling efficiency and generalization ability.

[0019] (2) A long short-term memory network is used to establish a temporal regression model from Mel spectrogram features to low-dimensional expression parameters, and the linear combination of the expression basis is driven by transparent and controllable intermediate parameters. This design breaks through the interpretability limitation of the end-to-end model, supports precise adjustment of expression parameters and avoids the "average face" effect, while ensuring the lip movement synchronization accuracy, enhancing the controllability and robustness of the generation process.

[0020] (3) Combining the mouth image generation driven by expression basis and the non-rigid deformation field technology, local geometric adaptation is achieved through the key point displacement field or three-dimensional vertex displacement, and the texture fusion is refined using the generative adversarial network. This mechanism takes into account both the facial detail fidelity and real-time performance while avoiding the high overhead of global rendering, and solves the contradiction between the low efficiency of traditional 3D methods and the lack of details in 2D methods.

[0021] (4) Through the speaker adaptation ability and the speech rate and rhythm adjustment mechanism, the output expression parameters are automatically adjusted for different speakers and different speech rates, ensuring the naturalness and accuracy of the digital human expressions under various voice inputs, and greatly improving the versatility and robustness of the system. In addition, the provided expression parameter interpretable adjustment module allows developers or the system to perform secondary fine-tuning on the generated expression intensity and style according to needs, and automatically constrains the overshoot or deficiency of the expression parameters in abnormal situations to avoid distortion. This interpretable and controllable expression adjustment ability makes the digital human expression generation both precise and flexible, combining technical rigor and practical application value.

[0022] (5) By decoupling the appearance modeling, audio driving and image fusion modules, a standardized data interface and a parallel pipeline architecture are constructed. Combining model compression, hardware acceleration and distributed computing technologies, it is adapted to multi-scenario deployments on mobile, cloud and web platforms. This framework breaks through the strong dependence of existing solutions on high-performance hardware, realizes high-frame-rate real-time driving under low-power devices, and significantly expands the application boundary of digital human technology.

[0023] (6) Only a single segment of the target person's video is required to clone and generate a digital human image, reducing data requirements and costs; by introducing semantic understanding and emotion driving mechanisms, the generated digital human facial expressions can match the speech content and the speaker's emotions, making them more vivid and realistic; adopting multi-modal alignment constraints improves the synchronization accuracy of the speech-driven lip movements and the audio; using self-supervised style consistency loss ensures the style stability and consistency of the generated digital human in different frame sequences, avoiding image flickering and distortion; and combining 3D semantic mask fusion improves the geometric matching degree and naturalness of face synthesis, thus significantly enhancing the realism and interactive experience of the digital human video as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] To make the above-mentioned single-video-based personalized digital human generation method of the present invention more clearly understandable, the drawings required for the specific implementation manners of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative efforts.

[0025] Figure 1 It represents the flowchart of the method of the present application; Figure 2 Schematic diagram of the video modeling module structure Figure 3 Schematic diagram of the speech feature extraction module structure Figure 4 Schematic diagram of the image fusion module structure Detailed implementation manners

[0026] Example 1 A personalized digital human generation method based on a single video is provided. The method process is as Figure 1 shown and is specifically described as follows.

[0027] 1. System module function logic and collaboration process This application includes the following three main functional modules. The logical functions of each module are as follows and are connected to each other to form a complete process.

[0028] The video modeling module is used to construct the basic model of the digital human's character image. The structure is as Figure 2 shown. This module takes a reference video of the target person or several static face images as input, extracts the facial features of the person for modeling. On the one hand, it uses face detection and key point extraction algorithms to obtain the geometric structure information of the person's face; on the other hand, it obtains the texture appearance information of the person's face. Based on the extracted features, a parametric facial model of the person is established, including identity parameters and an expression basis space. The identity parameters represent the inherent facial shape and appearance of the target person; the expression basis space is used to represent the form of facial dynamic changes and can be characterized by a set of basis functions obtained through pre-training to represent typical expression changes. Preferably, statistical modeling methods such as principal component analysis are used to offline train the low-dimensional basis of expression changes. For example, PCA modeling is performed on a large number of image sequences in the area of the person's mouth to obtain the average mouth image and several orthogonal basis features , which are used to represent the lip shape changes. In specific implementation, main basis vectors can be taken to summarize most of the mouth deformation features. The person appearance model output by this module includes: the neutral state face data of the target person, and the basis representation and mean value of expression / mouth shape changes. These data will be used as the basis for subsequent synthesis, so that the generated digital human has the same appearance features as the target person.

[0029] A deep learning model is used to train and model multiple expression samples of the target person. During the training process, a self-supervised style consistency loss function is introduced to ensure that the model maintains the consistency and stability of the style with the target person's image when generating the target face image.

[0030] The voice feature extraction module is used to convert the input voice signal into a parameter sequence for driving the facial expression change of the digital human, and the structure is as Figure 3 shown. This module first recognizes and transcribes the voice to obtain text, and performs semantic understanding through natural language processing technology to extract semantic information that conforms to the voice content; at the same time, it performs emotion recognition on the pitch, energy, and rhythm features of the voice signal to obtain an emotion label reflecting the speaker's emotional state. Then, the extracted acoustic feature parameters, together with the semantic information and emotion label, are input into the expression driving model to generate corresponding facial expression parameters to drive the facial model of the digital human to produce expression changes. It also includes a parallel audio emotion analysis model that classifies the emotion of the speaker in the voice by analyzing features such as pitch, volume change, and speech rate and outputs an emotion label. The output of the semantic and emotion analysis module will be transmitted to the expression generation module together with the voice features, enabling the expression synthesis process to perceive semantic and emotion factors. Specifically, it includes: performing voice feature processing on the input audio: dividing the original digital audio signal into short-time frames synchronized with the video frames, and calculating acoustic features such as Mel spectrogram coefficients for each frame to form an audio feature sequence , where represents the feature vector of the audio frame at the th moment. Then, the above audio feature sequence is input into a pre-trained voice driving model, and through temporal modeling and regression, the corresponding expression parameter sequence is output. Here, it is a low-dimensional expression parameter vector for controlling the digital human face at the The expression state of the frame video. The dimension of the expression parameter vector is consistent with the dimension of the expression basis space provided by the video modeling module, for example, preferably 6 dimensions. The audio-driven model is implemented using a deep neural network, for example, including a long short-term memory network layer to capture the phoneme persistence and co-articulation effects, and regressing the expression parameters at each moment through several fully connected layers. During the model training process, a large amount of data with face videos and corresponding voices is used as samples to minimize the error between the predicted mouth parameters and the real mouth parameters, so that the model learns the mapping relationship between audio and lip movement deformation. The model integrates a few-sample speaker adaptation mechanism, which can quickly adapt to new sound characteristics and expression habits by fine-tuning some network weights on a small amount of correction data of the new speaker, thereby maintaining the consistency of the driving effect between different characters; the model also has a built-in speech rate and intonation adjustment unit, which can dynamically adjust the change rate of the output expression parameters according to the real-time rhythm of the input voice, ensuring that the mouth shape changes at different speech rates are both sensitive and stable. After the expression generation module obtains the expression parameters, it applies the parameters to the digital human face model through the differentiable rendering layer to generate expression image frames. The rendering unit can call a graphics shader or a neural network to update the facial texture and maintain the consistency of the inter-frame and overall style through style constraints. At runtime, the module can quickly calculate a synchronized sequence of expression parameters based on any input speech, which will drive the movement of the digital human face in the next module. Through this audio-driven approach, the speech content can directly control the changes in the digital human's mouth shape and expression, ensuring that the generated digital human is strictly aligned with the input speech in terms of semantics and timing.

[0031] In order to ensure that the voice-driven facial movements are strictly synchronized with the audio signal, a synchronization mechanism based on multimodal alignment constraints is adopted in the process of expression parameter prediction. This mechanism uses the correspondence between the audio and the picture frame in the original video to correct the digital human generation process: when generating mouth-related expression parameters, the phoneme timing of the input voice is aligned with the rhythm of the target person's mouth movement to achieve accurate audio-visual synchronization. Through this multimodal alignment synchronization mechanism, the phenomenon of the voice-driven digital mouth shape and the sound being out of sync can be effectively avoided. In addition, a dynamic semantic feature control mechanism is introduced in the expression parameter mapping process: the value range of some expression parameters is dynamically adjusted according to the semantic stress and emotional intensity in the speech content. For example, when it is detected that the sentence contains a question tone, the expression parameters such as eyebrow raising are appropriately increased; when the emotional label is excitement, the parameter weight of the degree of smile is increased. Through this semantic / emotional linkage control, the generated expression not only matches the speech timing, but also echoes the speech content at the semantic level.

[0032] The image fusion module synthesizes continuous video frames based on the static face model provided by the video modeling module and the expression parameter sequence output by the speech feature extraction module, realizing the dynamic change of digital human facial expressions with speech signals. The structure is as follows:Figure 4 As shown. It is preferred to adopt a mask fusion and consistency checking mechanism based on three-dimensional semantic information. Specifically, three-dimensional semantic masks of key parts such as the mouth and around the eyes are obtained by using the pre-constructed 3D face model of the target person. When embedding the target facial local expression image generated by speech into the reference face image, the 3D semantic mask is used to define the fusion area, and the representational consistency of the fusion result is judged in combination with the three-dimensional semantic information. In this way, it is ensured that the inserted expression image highly coincides with the original face in terms of edge transition, scale, and orientation, and the overall expression of the fused face is natural and coordinated. Specifically, it includes: This module first initializes the reference image of the digital human. For example, the neutral expression face image output by the video modeling module is used as the initial base map, or the frontal reference face is rendered using the neutral face model. Subsequently, for each moment's expression parameters given by the speech feature extraction module, the image fusion module calculates the corresponding facial deformation and applies it to the reference image to generate a new frame. Specifically, in a preferred embodiment, the expression basis obtained in the appearance modeling stage is linearly combined to reconstruct the image of the target mouth area: Let be the mean image under neutral expression, be the -th frame's expression coefficient vector, then the reconstructed mouth image is expressed as: (1); In the formula, is the mouth area image of the -th frame; is the mean image under neutral expression; is the -th expression parameter of the -th frame; is the -th orthogonal basis vector in the expression basis space; is the number of expression basis vectors;

[0033] where each has the same dimension as and represents an orthogonal basis image in the mouth image space. Through the above linear combination, the mouth area image of the current frame can be quickly generated. Next, the generated mouth area is synthesized back into the whole face image: Based on the face key points obtained in the appearance modeling stage, the position and contour of the mouth area in the reference face base map are determined, and the reconstructed mouth image is used to replace the corresponding part in the reference face base map and perform stitching and fusion. For example, Image fusion techniques such as blending or seamless cloning ensure natural edge transitions and consistent colors. For expressions involving changes in facial geometric positions, the module also performs appropriate non-rigid deformations on the reference face image: using the key-point displacement field or the deformation field calculated from the 3D face model, a subtle distortion transformation is applied to the entire face, keeping parts such as the eyebrows and eyes consistent with the reference image, while the mouth area opens or closes according to the required shape. For example, a local affine transformation can be constructed based on the corresponding key-point pairs, or the method of triangulation interpolation can be used to deform the mouth polygon area of the neutral face into the target mouth shape of the current frame.

[0034] If a 3D face model is used to represent the expression basis, for each frame of expression parameters, first calculate the new set of 3D face vertex positions according to the parameters: (2); In the formula, is the set of 3D face vertex positions for the th frame; is the set of 3D face vertex coordinates in the neutral expression; is the th vertex displacement field corresponding to the expression basis, corresponding to different basis types in 2D and 3D models respectively. In this way, the geometric deformation of the face for the current frame can be obtained. Then, by calculating an appropriate 3D transformation, project the face shape onto the 2D screen coordinate system to obtain the new positions of the facial key points under this expression. Among them, the 3D rotation transformation is calculated using the Rodriguez formula: if is the unit vector of the rotation axis, is the rotation angle, then the rotation matrix is expressed as: (3); In the formula, is the identity matrix; is the skew-symmetric matrix composed of . By setting appropriate rotation axes and angles, simulate joint movements such as nodding, turning the head, or opening the mouth of the face. Use the projected new key-point coordinates to guide the mapping and interpolation of the 2D texture, and deform the texture of the reference face onto the current frame expression shape.

[0035] In any implementation path, the image fusion module finally obtains a sequence of frame images , where each frame is the synthesized facial image of the target person at the th time step. Continuously playing these frames can form a digital human video animation that is strictly synchronized with the input audio. By using pre-computed expression basis parameters, linear combinations, and efficient deformation processing, the module can quickly generate each frame of the image, ensuring the real-time nature of the generation process.

[0036] In addition, the expression parameters can be directly mapped through depth generation based on differentiable rendering to generate images of the entire human face or local expression regions. The differentiable rendering layer adopts a lightweight generative adversarial network or U-shaped network structure. Taking the neutral face of the target person as the initial condition, the learned rendering network performs image-domain conversion on the input expression parameters while keeping the person's identity characteristics unchanged, and outputs a face image corresponding to the expression. Specifically, during the training phase, an expression rendering sub-network is trained for the target person. This network takes the reference input of the baseline neutral face image and the expression parameters as the control signal, and outputs a rendered face image with an expression. The sub-network can adopt an Encoder-Decoder structure, and the expression parameters are fused in the Decoder to deform or modulate the feature layer, so as to generate expression changes corresponding to the input parameters. Due to the use of end-to-end learning, the rendering network can synthesize real images including complex changes such as frowning textures and lip details. To ensure that the rendering results are both realistic and maintain the consistency of the person's identity and style, multiple constraints are imposed on the rendering network during the training process: using content reconstruction loss to ensure that the synthesized expression is similar to the real expression image in pixels or features, using style transfer loss to ensure that the overall color texture style of the synthesized image is consistent with the baseline face, using adversarial loss to improve the realism of the generated image, and so on. Compared with the linear basis method, the differentiable rendering layer can depict richer expression texture details and allows end-to-end optimization training. When using differentiable rendering to generate, the present invention adds a style transfer constraint term to the loss function, and by calculating the difference in style features between the generated image and the original image of the target person, encourages the rendering network to generate an image effect consistent with the overall style of the target person.

[0037] The two generation methods can be combined: first, quickly give the initial expression synthesis through the linear basis, and then refine and improve the realism through the differentiable rendering network. The generated face expression image can be a complete face or only a local expression region, and will be fused with the baseline face according to needs later.

[0038] To sum up, through clear module division and rigorous algorithm design, this application realizes an efficient audio-driven digital human generation scheme. The input and output data structures of each module are clear, and the processing flow is consistent before and after. It can complete the conversion from speech to face animation with low computational complexity while ensuring the lip-sync accuracy and image realism, overcoming the dependence on expensive hardware devices and long training processes, and having the advantages of high usability, low resource occupancy, and strong real-time performance, which can fully meet the application requirements of real-time interaction scenarios such as virtual live broadcast, intelligent customer service, and online education.

[0039] The above-mentioned modules work together in a pipeline manner: after the video modeling module pre-establishes the static model and expression basis of the person, the speech feature extraction module maps the input speech to expression parameters in real time during operation and passes them to the expression generation module frame by frame; the image fusion module continuously updates the face image based on this and outputs the video stream. The data interfaces between the modules are clear: the reference image and expression basis parameters provided by the video modeling module are read and used by the latter two modules, and the expression parameters output by the speech feature extraction module are used as the input control quantity for image synthesis, ensuring seamless connection between speech driving and image generation. Through the close cooperation between the modules, the system as a whole realizes end-to-end conversion from audio to image with low latency, and the generated digital human facial expressions can accurately follow the changes in the audio content. The three modules jointly ensure the real-time and synchronization of digital human synthesis.

[0040] 2. System Execution Process The execution process of this method includes the following steps: Step 1: Obtain the person's material of the target character as the input, such as a 5-10 second video of the person speaking or several photos of different mouth expressions. Input the material into the video modeling module to extract the face features and construct the static model and expression basis library of the person. The obtained neutral face image and expression basis parameters will be saved in the system for subsequent steps to call.

[0041] Step 2: Receive the speech audio signal to be synthesized as the driving content. After preprocessing the audio, it is divided into continuous short-time frames and the acoustic feature parameters of each frame are extracted to form an audio feature sequence arranged in chronological order.

[0042] Step 3: Input the above audio feature sequence into the neural network model of the speech feature extraction module. The model processes the audio sequence step by step and outputs the corresponding expression parameter sequence . Whenever the audio accumulates to a duration equivalent to one frame of video, the model generates the expression coefficient at the current moment . In a simple implementation, the expression parameters can be directly regressed on each frame of audio features; in a preferred recurrent network implementation, the model will synthesize the audio features of several frames before and after to determine the expression parameters of the current frame to ensure the smoothness and accuracy of lip shape changes. The finally obtained expression parameter sequence is aligned with the audio time axis and enters the image synthesis preparation stage.

[0043] Step 4: Initialize the image fusion module and load the reference face image provided by the video modeling module as the background base map. Then for each set of expression parameters output by the speech feature extraction module , execute the following sub-steps to generate the corresponding video frame image: a. Linearly combine the expression basis according to the current expression parameters to calculate the synthesis result of the mouth area of the current frame ; b. Embed into the corresponding parts of the reference face image. If the 2D image method is adopted, replace the pixels in the mouth area of the base map and smooth the fusion edge; if the 3D mesh method is adopted, render the complete face image according to the deformed mesh. Optionally, perform additional adjustments to the head pose, and correspondingly translate or rotate the entire face image to obtain a coherent visual effect.

[0044] c. Output the currently synthesized face frame image to the video frame sequence cache and enter the processing of the next frame. The above process is repeated until the audio stream ends.

[0045] Step Five: After processing all audio frames, the image fusion module packages the generated sequential frame images in chronological order to form the output digital human video. The output video frame rate is usually equivalent to that of the reference person's video, and the resolution is the same as that of the reference face image. In the finally generated video, the lip movements and expressions of the digital human change naturally with the sound, achieving the required lip-sync effect.

[0046] Through the above steps, this method realizes a complete automated process from input audio to output digital human video. Among them, the appearance modeling belongs to an offline or preprocessing step and can be reused repeatedly after being completed once; the audio driving and image synthesis steps are executed in real time during operation, and the audio data and image frame data are processed in a parallel pipeline manner, ensuring that the overall system can generate digital human animations with low latency and high efficiency.

[0047] 3. Definition of Input and Output Data Structures To ensure clear module interfaces and consistent system implementation, professional definitions are made for the data structures of the main inputs and outputs of this application: The audio input is a single-channel digital voice signal, such as a.wav file or audio stream encoded in PCM. The preferred sampling rate is 16 kHz and the quantization precision is 16 bits, reducing the data volume while ensuring speech clarity. The audio data is organized in the form of a frame sequence during processing, and each frame corresponds to 20 - 40 ms of audio samples. A typical frame length is, for example, 512 sampling points, and the frames can be non-overlapping or have a certain overlap. After feature extraction, each frame of audio is represented as a feature vector containing several dimensions of real numbers, and these feature vectors are arranged in chronological order to form the audio feature sequence data structure for use by the driving model.

[0048] The inputs required for human appearance modeling include at least one video clip or photo set containing the face of the target person. Typically, RGB color image frames are used, and the resolution is selected according to application requirements. For example, to retain sufficient details, the size of the face region should be no less than 128×128 pixels. The input video can be represented as a sequence of image frames or in an encapsulated format. This data structure provides appearance samples of the target person with different mouth expressions. Through face key point extraction, the obtained face geometric features can be represented as a list structure of two-dimensional coordinates; if depth / 3D information is further included, it can be extended to a list of three-dimensional coordinates. The texture features of the person can use a reference face image. The output data of the video modeling module includes: the neutral face model of the target person and expression basis parameters. These data structures will be used as the basis for generating digital humans in subsequent processing.

[0049] Expression parameters are the key intermediate data output by the speech feature extraction module, and its structure is a list of vectors arranged in a time series. Each expression parameter vector corresponds to the facial expression state of one frame of video and is composed of several consecutive numerical values. Each numerical value represents the weight coefficient of a basic expression component. For example, each vector contains 6 floating-point numbers, representing the weights of the current frame's mouth shape on 6 main change bases. The data structure of this parameter sequence can be represented as a two-dimensional array of size where is the number of video frames, is the dimension of the expression parameters. If a 3D face parameterization model is adopted, the expression parameters can also include rigid motion components and non-rigid motion components, and the total dimension increases accordingly. The expression parameter data corresponds to the audio features and video frames one by one through time indexing and is the core control data for inter-module transfer and driving synthesis.

[0050] The finally output digital human video consists of a series of synthesized face image frames and has a time series synchronized with the input audio. The data format and resolution of each frame of image are the same as those of the input reference image. The frames are organized into a video sequence at a fixed frame rate and stored as a standard video file. The output video data structure includes: a list of frame image pixel matrices , the number of frames and frame rate information, as well as the corresponding audio track data. In the case of only focusing on the visual output, only the image sequence is output. In practical applications, the output results are directly presented frame by frame on a display device or used for storage and transmission after encoding and compression. In any form, the output data structure ensures that the facial movements of the digital human are perfectly aligned with the input audio content and provides the final effect of real-time synthesis of the digital human to the outside.

[0051] 4. Cross-platform applicability and exception handling This method and system have good cross-platform applicability and can be deployed to run in different hardware environments. In the mobile environment, due to the lightweight and efficient algorithm model, the CPU or mobile GPU of the mobile device can be used to execute each module at real-time speed. The appearance modeling and audio-driven models are pre-computed on the server or offline, and the obtained model parameters are packaged into the mobile application, enabling the mobile device to only perform real-time inference calculations. For deep neural network models, the volume is reduced through model compression and quantization, and a neural network acceleration framework for mobile devices is used to accelerate inference, thus enabling smooth digital human synthesis on ordinary mobile phone chips. In the web environment, this system executes key algorithm modules in the browser through technologies such as WebAssembly or WebGL. For example, the audio feature extraction and part of the expression prediction models are deployed as JavaScript / WebAssembly scripts to run on the front end, and the expression generation module uses WebGL shaders or GPU acceleration libraries for image processing. For steps with higher computing requirements, a front-end and back-end collaboration method is adopted: after the user's audio is obtained on the browser side, it is sent to the server for processing, and then the generated face frame stream is returned to the web page for real-time playback. In the cloud server, due to having sufficient computing and storage resources, this solution supports higher-resolution video synthesis and concurrent requests from more users. Each module is distributedly deployed in the back-end cluster, and the user-requested audio is processed by calling the pre-loaded deep learning model instance, and the corresponding video frame sequence is quickly generated. With the help of GPU acceleration in the cloud, this system can further reduce latency while ensuring high fidelity, and provide real-time digital human driving services for a large number of users simultaneously through load balancing.

[0052] To improve the robustness of the system under various input conditions, corresponding abnormal scenario handling strategies are also designed. When the audio is missing or interrupted, the system automatically keeps the digital human's face on the neutral expression of the previous frame or remains stationary, and only resumes lip driving after receiving audio input again, thus avoiding meaningless lip movements of the digital human when there is no audio. When the audio noise is too large or does not meet expectations, the audio feature extraction module combines silence detection and noise filtering algorithms. When a long silence or invalid audio segment is detected, the expression parameters of the corresponding frame are smoothly transitioned or maintained at neutral to ensure the stability of the synthesis result. When face key point extraction fails, the system enables an alternative solution: for example, using an average face model instead, or prompting the user to provide clearer input materials. When abnormal values appear in the expression parameters during operation, the expression generation module will perform constraint clipping and interpolation smoothing on the parameters to prevent the generation of deformed facial images. For streaming scenarios such as real-time camera input, in case of frame loss or delay, the system ensures the synchronization of audio and video processing through buffer and interpolation strategies, and even if individual frames are missing, it will not significantly affect the lip coherence of the digital human. Through the above measures, this system can operate stably under various abnormal conditions and provide reliable digital human synthesis output.

[0053] 5. Application Scenarios This system can be widely applied to various actual scenarios, significantly reducing the threshold for digital human video production. For example, in the field of virtual live streaming, it is used to create real-time broadcast content of virtual anchors: simply providing a short video sample and real-time audio of the anchor, this system can generate the lip movements and expressions of the virtual anchor synchronized with the audio, enabling online live performances or news broadcasts and reducing the cost of real people appearing on camera. Another example is in the digital human customer service scenario, where it can be used to build an intelligent customer service with a humanized image. The appearance materials of the customer service representative are pre-recorded and the model is deployed. When the background customer service AI generates a voice response, a video response corresponding to the customer service image is synchronously generated, providing an immersive interaction experience for users. Another example is in online education, where it can be used to produce virtual digital teachers or training anchors. As long as the teacher provides a video recording and course voice once, this system can automatically synthesize a video of the teacher's image giving a lecture, which is used for large-scale online course content production. In addition, the present invention is also applicable to other scenarios that require converting voice content into a video with lifelike lip synchronization, such as film and television dubbing, game character dialogues, and virtual avatars on social media. In these applications, this system can achieve high-quality digital human driving effects at low cost, meeting the requirements of real-time interaction or batch content production.

[0054] Example 2: Based on the foregoing embodiment, the system is applied to the mobile virtual anchor scenario. Suppose a user hopes to generate their own digital human anchor image on the mobile phone and drive its lip movements with real-time voice. First, the user uses the mobile phone camera to record a video of about 10 seconds. In the video, the user reads a few sentences with a natural neutral expression to provide lip movement materials. This video is input as a reference into the video modeling module, which automatically detects the human face in the video and extracts the coordinate sequence of 68 facial key points and a clear neutral expression human face image. Using the PCA lip base pre-trained offline, the video modeling module establishes a static model of the user's human face and a lip expression base library.

[0055] Next, the system loads the pre-deployed audio driving model and image fusion module on the mobile phone. The user inputs voice through the mobile phone microphone or selects a piece of text to be synthesized into voice by the TTS engine. The voice feature extraction module extracts Mel spectrogram features frame by frame from the real-time collected voice signal, and runs a lightweight LSTM network model on the mobile phone CPU to gradually predict and output the sequence of expression parameters. As the voice is input, the system continuously generates the mouth expression parameters for the next frame. The image fusion module uses the pre-acquired neutral face image of the user as the base map, and performs generation and fusion for the mouth area of each frame on the mobile phone GPU: generates a mouth image segment through linear combination according to the current expression parameters, pastes it onto the neutral face base map of the user, and performs smooth transition processing on the edges. When the user's speech rate is relatively fast, the system automatically predicts the expression parameters slightly in advance to compensate for the processing delay and ensure that the generated frames are strictly aligned with the sound signal.

[0056] Finally, the user sees their digital human avatar on the mobile phone screen opening and closing its mouth and showing corresponding expressions along with the voice content, achieving a real-time virtual anchor effect. During the whole process, the calculation on the mobile phone side is smooth without obvious lags. The actual test achieves a synthesis frame rate of about 25 frames per second on an ordinary smart phone, verifying the real-time applicability of this system on mobile devices.

[0057] Embodiment 3: Based on the foregoing embodiments, this system is deployed on a cloud server to build a digital human customer service interaction system. Suppose a certain platform hopes to use a digital human to undertake the role of online customer service on the web page. First, it is necessary to prepare the materials for the customer service digital human: select an employee with an amiable image and record a short video of this employee reading the customer service welcome message as a reference for the character appearance. Input this video into the video modeling module to extract the static model data of the employee's face. These model data are stored on the server side as the basic image of the digital human customer service. Subsequently, the trained voice driving model and image synthesis model are loaded at the server backend. When a website user initiates a conversation consultation, the text-to-speech module of the customer service dialogue system synthesizes voice according to the text content to be replied and sends this audio to the digital human synthesis server.

[0058] After the system receives the audio, the speech feature extraction module immediately extracts features from the audio and predicts the expression parameters to obtain a complete sequence of expression parameters. Then, the expression generation module uses the pre-stored neutral face image of the customer service as the base map and generates the facial images of the customer service speaking with open mouth frame by frame: for each expression parameter in the sequence, it generates the corresponding mouth shape image and fits it onto the neutral face. If the customer service's reply is long, the system receives the audio stream while outputting the image frame stream in real time, making the mouth shape of the digital human change continuously. All the generated frames are packed into a video stream at a frame rate of 25 frames per second in the cloud and pushed to the front end of the user's browser through WebSocket to be played synchronously with the synthesized voice, presenting a virtual customer service image with a tightly corresponding mouth shape and voice. The user can see the digital human customer service smiling in the browser and the mouth shape accurately speaking the reply content, and the effect is close to that of a real person. Even if there is a little delay in network transmission, due to the high efficiency of the system generation process, the overall interaction still maintains a response time in the order of hundreds of milliseconds, ensuring a good user experience. It proves the feasibility of the system in the cloud service mode and can support large-scale real-time interactive digital human applications.

[0059] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable non-transitory storage media containing computer-usable program code.

[0060] The present invention can provide computer program instructions to the management platform of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed through the management platform of the computer or other programmable data processing devices generate the device for implementing the system.

[0061] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions of the system.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide the steps for implementing the functions of the system.

Claims

1. A personalized digital human generation method based on a single video, characterized in that, Including: Obtain a video or several images of a target person, and construct a digital human facial model of the target person. During the model training process, introduce a self-supervised style consistency loss to ensure the stable consistency of the generated style; Obtain the speech input of the target person and extract acoustic feature parameters; Conduct semantic understanding and emotion recognition on the speech input to obtain corresponding semantic information and emotion labels; Under the constraint of multi-modal alignment, map the acoustic feature parameters and the semantic information and emotion labels to corresponding facial expression parameter sequences; among them, a dynamic semantic feature control mechanism is introduced for the semantic stress and emotion intensity of the speech content to adjust the amplitude and details of the generated expressions; According to the facial expression parameter sequence, generate a face image of the target person through the digital human facial model; the generation process includes a differentiable rendering layer, performs end-to-end expression synthesis on key facial regions, and combines style transfer constraints to ensure that the generated local expression images are consistent with the overall appearance style of the target person; Use a 3D semantic mask to fuse the face image of the target person into a pre-stored reference face image of the target person, complete the fusion and superposition of the local expression area and the reference face at the pixel level, and output a personalized digital human video synchronized with the speech input content.

2. The method according to claim 1, wherein: The speech input is preprocessed and divided into continuous short-time frames, and Mel spectrogram coefficients or MFCC coefficient acoustic features are extracted. The acoustic features are input into an audio-driven model of a deep neural network to regress and output a facial expression parameter sequence; the audio-driven model includes a long short-term memory network layer for capturing the temporal dependence of the speech signal, and further includes a speaker adaptation mechanism for fine-tuning some parameters of the model under the condition of only providing a small amount of speech-expression correspondence data of the target speaker, so as to adjust the expression parameter mapping to fit the pronunciation style and expression habits of this speaker.

3. The method according to claim 2, wherein: The audio-driven model further includes an adjustable mapping unit for different speaker speech rates and tone rhythms. The mapping unit dynamically adjusts the output rate and smoothness of the facial expression parameter sequence according to the phoneme duration, speech rate, and intonation fluctuations of the speech signal; By introducing additional features or modules reflecting the speech rhythm, ensure a fast response and stable transition of the generated mouth movements at a fast speech rate, and maintain smooth and natural expression changes at a slow speech rate.

4. The method according to claim 2, wherein: The audio-driven modeling process can also adopt a Transformer network, a temporal convolutional network or other equivalent alternative structures.

5. The method according to claim 1, wherein: The process of using the semantic information and emotion labels to drive expression generation includes a context semantic linkage mechanism for dynamically adjusting additional facial expression features such as eyebrows and eye expressions according to the semantic focus and emotion category in the speech content.

6. The method according to claim 1, wherein: The process of generating a target facial expression image from a digital human facial model includes using a differentiable rendering layer to perform expression synthesis rendering on local facial regions; the differentiable rendering layer receives the sequence of facial expression parameters and combines it with the target person's facial model to generate corresponding frames of the target person's expression image.

7. The method according to claim 1, wherein: The use of a 3D semantic mask for image fusion specifically includes: Step 1: Based on a pre-constructed three-dimensional facial model of the target person, extract the three-dimensional semantic masks of the mouth and the area around the eyes; Step 2: Dynamically adjust the geometry and spatial position of the mask according to the expression parameters of the current frame to make it adapt to the opening and closing state of the target mouth shape; Step 3: Perform pixel-level blending of the generated target local facial expression image and the pre-stored reference facial image within the fusion area defined by the 3D semantic mask, and verify the alignment consistency of the orientation and scale of the fusion area and the reference face through a spatial transformation matrix.

8. A personalized digital human generation system based on a single video, wherein: When the system runs, it executes the method according to any one of claims 1-7: The system includes: A video modeling module for constructing a digital human facial model of the target person based on a video or several images of the target person, and introducing a self-supervised style consistency loss during model training to ensure stable and consistent generation style; A voice feature extraction module for extracting the acoustic feature parameters of the voice input of the target person; A semantic and emotion analysis module for performing semantic understanding and emotion recognition on the voice input to obtain corresponding semantic information and emotion labels; An expression generation module for mapping the acoustic feature parameters and the semantic information and emotion labels to corresponding facial expression parameters under multi-modal alignment constraints, and generating a face image of the target person according to the facial expression parameters; An image fusion module for fusing the face image of the target person into a pre-stored reference face image of the target person using a 3D semantic mask, and outputting a personalized digital human video.

9. The system according to claim 8, wherein: The voice feature extraction module further supports multi-modal input, including using the audio generated by text-to-speech as a driving signal to achieve an end-to-end mapping from text to digital human expression.

10. The system according to claim 8, wherein: It further includes an expression parameter interpretable adjustment module for constraining and refining the adjustment of the sequence of facial expression parameters; when the voice input is interrupted, the adjustment module maintains the current expression parameter state or makes the expression parameters smoothly transition to the neutral value; when any expression parameter exceeds the preset reasonable range, the adjustment module adjusts it within the threshold by means of clipping limitation or interpolation correction; the module also provides several interpretable control indicators for manual or algorithmic real-time adjustment of the expression intensity and style.

Images