A virtual digital population broadcasting optimization method and device

Abstract

This application discloses a method and apparatus for optimizing virtual digital human voice broadcasting, comprising: acquiring facial image data and speech signals of a specific person, and extracting a final facial feature vector; generating a personalized oral cavity internal structure description that conforms to the anatomical characteristics of the specific person based on the final facial feature vector; generating a speech driving parameter sequence for the speech signal to be broadcast, in conjunction with the personalized oral cavity internal structure description; processing the speech driving parameter sequence to generate a mouth movement trajectory that coordinates with the personalized oral cavity internal structure, and generating an oral cavity animation based on the trajectory; integrating the oral cavity animation with the overall facial animation, and optimizing identity consistency through a closed-loop feedback mechanism. This method can generate a dynamic oral cavity model that conforms to the anatomical characteristics of a specific person, effectively improving the personalization level, identity consistency, and naturalness of virtual digital human animation.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer graphics, specifically to a method and apparatus for optimizing virtual digital population broadcasting. Background Technology

[0002] Virtual digital human technology, a product of the cross-disciplinary integration of artificial intelligence, computer graphics, and virtual reality, has been widely applied in scenarios such as virtual anchors, online education, film and television production, and customer service. This technology aims to provide an immersive human-computer interaction experience by generating virtual avatars with realistic appearances, expressions, and behaviors through computer processing. However, as users' demands for realism and personalized expression in virtual digital humans continue to increase, existing technologies still face numerous technical challenges in generating high-quality virtual digital human animations with specific identity characteristics.

[0003] On the one hand, existing methods for generating virtual digital humans often struggle to accurately reproduce the unique physiological characteristics of a specific person when dealing with facial animation, especially lip animation. Many technical solutions rely on generic facial topology models or pre-set animation template libraries, resulting in virtual digital humans exhibiting homogenization and stereotyping in facial expressions and lip movements, lacking the personalized characteristics of the target person. This is particularly true when portraying complex speech content, making it difficult to reflect the person's identity. Viewers watching such virtual digital human narrations easily perceive a mechanical and distorted feel in the animation, thus affecting the immersive experience of the interactive experience.

[0004] On the other hand, generating highly realistic mouth animations requires precise modeling and driving of the dynamic structures inside the oral cavity. Existing technologies mostly focus on driving mouth shape changes through external facial features (such as lip key points), while simplifying or completely ignoring the anatomical structures inside the oral cavity, such as the shape and arrangement of teeth, the specific shape of the tongue, and its complex movement patterns. Due to the lack of personalized modeling of these internal structures, the generated animations, when depicting actions such as opening and closing the mouth to speak, may appear unnatural or inconsistent with the real physiological characteristics of the target person. For example, when a virtual digital human is performing continuous narration, its mouth movements may not be completely synchronized with the phonemes and rhythm of the speech, or the movements of the teeth and tongue may not conform to the anatomical characteristics that the person should have, leading to visual incongruity and weakening the identity consistency of the virtual digital human animation. Identity consistency not only requires the virtual character to resemble the target person in appearance, but also requires it to reflect the uniqueness of a specific person in subtle details such as micro-expressions and movement habits. Therefore, how to generate a dynamic oral cavity model that conforms to the anatomical features of a specific person based on only limited facial information, and ensure that its animation and voice input are highly matched, has become a technical problem to be solved in order to improve the realism and identity consistency of virtual digital humans.

[0005] Furthermore, in special scenarios where the accuracy, timeliness, and clarity of information transmission are extremely important, such as emergency command, public broadcasting, and disaster reporting, human broadcasters may experience emotional fluctuations, slips of the tongue, or omissions due to tension, fatigue, or environmental noise interference, which could affect the reliability of instruction delivery. Therefore, developing a virtual digital human technology capable of maintaining stable, accurate, and clear broadcasting capabilities in such high-pressure and noisy environments has significant application value. Summary of the Invention

[0006] One aspect of this invention is to provide a virtual digital human voice broadcasting optimization method to solve the technical problems mentioned in the background art, such as insufficient personalization of virtual digital human animation, poor identity consistency, and low reliability in critical information transmission scenarios.

[0007] To achieve the above objectives, the present invention provides a virtual digital population broadcasting optimization method, comprising the following steps:

[0008] Step 1: Acquire facial image data of a specific person and the corresponding voice signal. Process the facial image data to extract preliminary facial features, and combine them with the lip movement synchronization information extracted from the voice signal to form a final facial feature vector.

[0009] Step 2: Based on the final facial feature vector, generate a personalized description of the oral cavity structure that reflects details such as the tooth arrangement and tongue shape of the specific person.

[0010] Step 3: Acquire the speech signal to be played, analyze the speech signal to generate a preliminary driving parameter sequence, and then combine the constraints of the personalized oral cavity internal structure description to optimize the parameter sequence and map it into an oral movement trajectory that is coordinated with the structure.

[0011] Step four: Based on the oral movement trajectory data, generate an oral cavity animation containing detailed internal dynamics, and synthesize it with the overall facial animation data to generate a preliminary virtual digital human animation;

[0012] Step 5: By extracting facial key point sequences from the preliminary virtual digital human animation and the real reference video of the specific person, respectively, and calculating the dynamic time warping (DTW) distance between the two sequences, a quantitative identity consistency index is obtained.

[0013] Step six: If the identity consistency index exceeds the preset range, the index is used as the optimization target, the personalized oral cavity internal structure description is updated retrospectively, and the subsequent steps are repeated until the index converges to the preset range, so as to output the final virtual digital human animation.

[0014] On the other hand, the present invention also provides a virtual digital voice broadcast optimization device for performing the above-described method, comprising:

[0015] The data processing module is used to acquire facial image data of a specific person and synchronized speech signals, process the facial image data to extract preliminary facial features, and combine them with lip movement synchronization information extracted from the speech signals to fuse them into a final facial feature vector.

[0016] The structure generation module is used to generate a personalized description of the internal oral structure that reflects details such as the tooth arrangement and tongue shape of the specific person based on the final facial feature vector.

[0017] The trajectory generation module is used to acquire the speech signal to be played, analyze the speech signal to generate a preliminary driving parameter sequence, and then, in combination with the constraints of the personalized oral cavity internal structure description, optimize and map the preliminary driving parameter sequence into oral movement trajectory data that is coordinated with the personalized oral cavity internal structure description.

[0018] An animation synthesis module is used to generate an oral cavity animation containing detailed internal dynamics based on the oral movement trajectory data, and synthesize it with the overall facial animation data to generate a preliminary virtual digital human animation.

[0019] The feedback optimization module is used to extract facial key point sequences from the initial virtual digital human animation and the real reference video of the specific person, calculate the dynamic time warping (DTW) distance between the two sequences, thereby obtaining a quantified identity consistency index. When the identity consistency index exceeds a preset range, the index is used as the optimization target to retrospectively update the personalized oral cavity internal structure description, and the subsequent steps are repeated until the index converges to the preset range to output the final virtual digital human animation.

[0020] Compared with existing technologies, the beneficial effects of this invention are as follows: By employing a deep learning-based method, it directly generates personalized oral cavity structure descriptions containing details such as teeth and tongue from facial image data, overcoming the problem of insufficient personalization caused by the reliance on general templates in traditional methods. This allows for the generation of dynamic oral cavity models that conform to the specific anatomical characteristics of individuals, effectively improving the personalization level and identity consistency of virtual digital human animations, making mouth movements more natural and realistic. Furthermore, the closed-loop feedback optimization mechanism based on identity consistency indicators proposed in this invention can continuously fine-tune the generated oral cavity structure model, ensuring high fidelity in animation output. In particular, by integrating noisy environment adaptation processing, emotional fluctuation avoidance mechanisms, and robustness enhancement strategies for key information transmission in extreme environments, the virtual digital human generated by this method can maintain stable, clear, and accurate speech delivery in high-pressure, noisy environments such as emergency command and public broadcasting, effectively solving the problem of insufficient reliability of existing technologies in such special application scenarios. Attached Figure Description

[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart of a virtual digital voice broadcasting optimization method provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram illustrating the generation of a personalized oral cavity internal structure description in an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of a closed-loop feedback optimization mechanism based on the identity consistency index in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of a strategy for enhancing the robustness of key information transmission in extreme environments, as described in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram illustrating the effect of the emotion fluctuation avoidance mechanism in smoothing mouth movements in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the construction of a personalized three-dimensional oral cavity structure in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0029] Example 1

[0030] This embodiment provides a virtual digital voice broadcast optimization method, such as... Figure 1 As shown, this method is used to generate highly personalized and identity-consistent virtual digital human animations, enhancing their information transmission reliability in specific scenarios. This method can be applied to computer devices, servers, or cloud platforms, and is implemented by executing corresponding program instructions. The specific steps of this method will be described in detail below.

[0031] Step S1: Acquire and process data of a specific person to extract high-precision facial features.

[0032] First, multiple frames of facial images of a specific person speaking naturally are captured using a camera device, and their voice signal is recorded simultaneously. After acquiring the data, it is preprocessed, such as denoising the images and normalizing the illumination.

[0033] Specifically, a pre-trained convolutional neural network (CNN) is used to process the pre-processed multi-frame facial image data. The CNN network structure can employ mature architectures such as ResNet or VGG, which have been trained on large datasets of faces and can automatically learn and extract deep features from the images. In this step, the network outputs a preliminary facial feature vector, which encodes static and quasi-static features such as the precise location of the facial contours, the geometric relationships of the facial features, and skin texture information.

[0034] Subsequently, to enhance the expressive power of the dynamic features of the lip region, information from the synchronized speech signal needs to be introduced. Lip movement synchronization information, such as the energy envelope and phoneme timing, is extracted from the synchronized speech signal. Using methods such as cross-correlation calculation, the energy change trend of the speech signal is temporally aligned and compared with the motion trajectories of key lip points (such as the corners of the lips and the center points of the upper and lower lips) in the image sequence, thereby calculating a quantified matching degree between lip movement and speech. Based on this matching degree, a lip movement enhancement vector is generated, which can characterize the uniqueness of lip movements during pronunciation for a specific individual.

[0035] Next, the initial facial feature vector is fused with the lip movement enhancement vector, for example, through vector concatenation or weighted summation, to construct a comprehensive feature matrix. To optimize this matrix, eliminate redundant information, and extract core features, the linear correlation between features is evaluated by calculating the Pearson correlation coefficient between the elements in the matrix. For a pair of features with a correlation higher than a preset threshold (e.g., 0.9), one can be retained or they can be merged. After this step, an optimized feature representation is formed.

[0036] Finally, the optimized feature representation is evaluated for information entropy to determine its information complexity. If the information entropy meets the preset complexity threshold, it indicates that the feature dimension is moderate and the information content is rich. If the information entropy is too high, it may mean that there is redundancy or noise. In this case, principal component analysis (PCA) is used to reduce its dimensionality, selecting principal components with a contribution rate reaching a predetermined percentage (e.g., 95%), thereby obtaining a final facial feature vector that can both retain core identity information and has a moderate dimension for subsequent steps.

[0037] Step S2: Generate a personalized description of the internal structure of the oral cavity based on the final facial feature vector.

[0038] This step involves constructing a 3D model of the oral cavity that conforms to the specific anatomical features of the individual, including details such as teeth and tongue. For example... Figure 2 As shown, this embodiment preferably adopts a generation scheme based on a three-dimensional conditional generative adversarial network (3D ConditionalGAN). Figure 6 The process of constructing a personalized three-dimensional structure of the oral cavity using 3D conditional generative adversarial networks is demonstrated in detail.

[0039] The final facial feature vector extracted in step S1 is used as conditional input and fed into a pre-trained 3DGAN generator. This network is trained on a dataset containing a large amount of 3D oral cavity scan data, which can be obtained through medical imaging techniques such as computed tomography (CT) or magnetic resonance imaging (MRI), ensuring the anatomical accuracy of the generated model. Based on the input external facial features, the generator can directly generate a novel, high-fidelity 3D mesh model of the oral cavity. This model can accurately reflect the personalized anatomical structures of a specific individual, such as the details of tooth arrangement (e.g., sparseness, crowding, and tilt), tooth morphology (e.g., size and shape of incisors), and tongue size and shape.

[0040] Subsequently, to make the model more biomechanically consistent during animation, skeletal support vectors for supporting facial muscle movements were extracted from the 3D mesh model of the oral cavity. These vectors were obtained by identifying and parameterizing the geometry of regions representing the maxilla and mandible in the 3D mesh model, defining the physical boundaries and fulcrums of mouth movements. This skeletal support vector was then fused with the 3D mesh model, for example, by using it as an additional attribute or constraint, to construct an enhanced oral cavity model with more realistic mechanical properties.

[0041] Finally, the structural stability of the enhanced oral cavity model is evaluated. The evaluation can be conducted in a physical simulation environment by applying simulated muscle traction to the model and observing its deformation under simulated vocalization movements. If the maximum vertex displacement is below a preset deformation threshold, it is determined to be the final personalized description of the oral cavity's internal structure. Otherwise, the deformation error is used as a feedback signal to fine-tune the generator network parameters and regenerate the model until its stability meets the target.

[0042] Step S3: Analyze the speech signal to be played and generate a parameter sequence to drive the mouth animation.

[0043] After obtaining the personalized oral cavity model, this step generates parameters that can drive the model to produce corresponding oral animations based on any input speech signal to be played.

[0044] First, the input speech signal to be broadcast is preprocessed, and its rhythmic features are extracted using signal processing techniques, such as calculating speech rate and identifying pauses by analyzing short-time energy and zero-crossing rate; at the same time, it is converted into a phoneme sequence using an acoustic model.

[0045] Then, a recurrent neural network (RNN), preferably a long short-term memory network (LSTM) or a gated recurrent unit (GRU), is used to model the rhythmic features and phoneme sequences containing temporal dependencies. The structural characteristics of RNNs enable them to effectively capture the dynamic characteristics of speech that change continuously over time. This network is trained to learn the mapping relationship from speech features to lip shape parameters, and its output is a preliminary sequence of driving parameters. This sequence is typically a time series, with each frame containing multiple parameters, such as the degree of mouth opening, the degree of lip contraction, and the position of the tongue.

[0046] To better align the driving parameters with the vocal habits of a specific individual, the personalized oral cavity structure description obtained in step S2 is used as constraint information. A parameter vector weighting adjustment module optimizes the distribution of the initial driving parameters. For example, if the personalized model indicates a large oral cavity space, the weights of parameters related to mouth opening are increased accordingly; if the tongue is large, parameters related to the range of tongue movement are adjusted. In this way, an enhanced driving sequence is determined, whose parameter distribution better matches the individual's physiological structure. Finally, this enhanced driving sequence is used to optimize the mapping relationship between phonemes and corresponding lip shape parameters during phoneme dynamic modeling, generating a speech driving parameter sequence.

[0047] Step S4: Process the speech-driven parameter sequence to generate a coordinated mouth movement trajectory and incorporate environmental adaptability.

[0048] This step transforms the driving parameters into specific three-dimensional spatial motion trajectories, ensuring they are smooth, natural, and adaptable to noisy environments.

[0049] First, the temporal information corresponding to key phonemes (such as vowels a, i, u and plosives b, p) is extracted from the speech driving parameter sequence. To analyze their dynamic characteristics more precisely, Fourier transform is used to convert this temporal information from the time domain to the frequency domain, thereby obtaining its frequency features. These features reflect information such as the velocity and acceleration of mouth movements. By calculating the correspondence between frequency peaks and the original temporal points, a temporal frequency distribution is obtained.

[0050] Then, a pre-trained deep neural network (DNN) is used as a mapping function to combine the temporal frequency distribution with the personalized oral cavity internal structure description obtained in step S2. Specifically, a quantified coordination value is obtained by calculating the inner product of the vectors of each frequency point in the temporal frequency distribution and the vectors of key oral cavity points (such as the corners of the lips, the tip of the tongue, and the edges of the teeth) in the structural description. This inner product operation measures the consistency in direction between the vector representing the movement trend and the vector representing the structural position, thereby quantifying whether the dynamic movement conforms to the physical constraints of the static structure and reflecting the degree of matching between the dynamic characteristics of the movement and the static structure. If the coordination value is lower than a preset threshold, it indicates that the current mouth movement may exceed the reasonable range of the person's physiological structure. At this time, the network weights of the mapping function of the DNN are adjusted through the backpropagation algorithm until the coordination value calculated by the output coordinated temporal structure data reaches the preset threshold.

[0051] Furthermore, this invention enhances animation smoothness by incorporating noisy environment adaptation processing. Specifically, noise components are extracted from the environmental audio signal acquired synchronously with the speech signal using spectral subtraction or a deep learning-based denoising model to generate an environmental noise spectrum. Subsequently, a Kalman filter is used to fuse the interference information represented by the environmental noise spectrum with the coordinated temporal structure data. The Kalman filter can predict the trajectory state at the next moment based on the system's dynamic model and make corrections by incorporating noisy observations (i.e., coordinated data). The environmental noise spectrum is used to dynamically adjust the measurement noise covariance matrix in the filter. When the environmental noise is high, the value of this matrix is ​​increased, making the filter rely more on the predicted value, thereby effectively filtering out trajectory jitter caused by random noise interference and determining a smoother trajectory. Mouth motion trajectory data is generated based on this smoother trajectory, and its smoothness is evaluated. If the variance of the second derivative of the trajectory (i.e., acceleration) exceeds a preset threshold, the process noise parameter Q and measurement noise parameter R of the Kalman filter are iteratively adjusted to finally obtain smooth and coordinated mouth motion trajectory data.

[0052] Step S5: Generate detailed oral cavity animation and integrate a stability assurance mechanism.

[0053] Based on the coordinated mouth movement trajectory data obtained in step S4, the final animation clip is generated.

[0054] Specifically, based on the mouth movement trajectory data, the 3D position and pose sequences of the teeth and tongue in each frame of the animation are analyzed. These sequences are then applied to the personalized oral model generated in step S2 to generate an initial animation frame sequence containing precise movements of the teeth and tongue.

[0055] To enhance the visual effect, sharpness optimization is applied to the initial frame sequence. For example, an unsharp masking algorithm can be used. This algorithm creates a mask by subtracting the Gaussian blurred version of the original image, and then adds the mask back to the original image at a certain ratio. This enhances the edges and details of the image at the pixel level, making the edges of teeth and the texture of the tongue clearer and more realistic.

[0056] Furthermore, this invention also integrates an emotion fluctuation avoidance mechanism, which effectively smooths the mouth movement trajectory as follows: Figure 5 As shown, this mechanism ensures the emotional stability of the virtual digital human during serious tasks such as emergency broadcasts. It identifies prosodic features such as pitch (fundamental frequency), speech rate, and energy in the input speech signal in real time, pinpointing prosodic feature peaks that may be triggered by emotions like tension or excitement. When generating the animation frame sequence, a smoothing filter, such as a Gaussian filter, is applied to the mouth movement trajectory at the time points corresponding to these emotional peaks to appropriately suppress unnatural or exaggerated lip movements that may occur due to emotional fluctuations, ensuring the professionalism and authority of the broadcast. Finally, based on the adjusted parameters, an animated mouth segment integrating clarity optimization and emotional stability is generated.

[0057] Step S6: Integrate the animation and optimize identity consistency and information reliability through closed-loop feedback.

[0058] This step integrates the generated oral cavity animation with the animations of other parts of the face and introduces a feedback mechanism for global optimization. For example... Figure 3 As shown, a closed-loop optimization mechanism based on the identity consistency index is constructed.

[0059] First, the oral cavity animation clips and preset overall facial animation data are acquired, the latter including blinking, eyebrow movements, head posture, etc. The oral cavity animation frame sequence and the facial animation sequence are then synthesized using timecode alignment to generate a preliminary virtual digital human animation.

[0060] To quantitatively assess identity consistency, facial keypoint sequences were extracted from the initial animation, and the same keypoint sequences were extracted from a real reference video of the specific person. A quantitative identity consistency metric was obtained by calculating the Dynamic Temporal Warping (DTW) distance between the two sequences. The DTW algorithm effectively measures the similarity between two time sequences of different lengths or rates, and is suitable for evaluating the realism of motion imitation.

[0061] If the identity consistency index exceeds a preset range, it indicates a significant deviation between the generated animation and the behavior patterns of a real person. In this case, the index is used as part of the loss function, and optimization algorithms such as gradient descent are used to backtrack and update the relevant parameters of the personalized oral cavity structure description generated in step S2. For example, the conditional vector input to the 3D cGAN can be adjusted, or the network weights of the 3D cGAN generator can be fine-tuned. Then, steps S2 to S6 are re-executed, forming a closed loop, until the index converges to the preset range. If the index is within the preset range, the current animation is determined to be the final identity-consistent virtual digital human animation output.

[0062] Before output, an information verification mechanism is further applied. This mechanism accurately compares the phoneme sequence corresponding to the output animation (which can be deduced through lip-sync recognition) with the original phoneme sequence of the input speech signal. If a mismatch or omission of a phoneme is found, the animation at that location is automatically marked and a correction process is triggered, such as locally regenerating the lip-sync animation for that part, to eliminate mispronunciations and omissions, ensuring the reliable transmission of critical information such as emergency instructions.

[0063] In a preferred embodiment, to meet the higher requirements for model reusability and clarity of information transmission in noisy environments in specific scenarios such as emergency command and public broadcasting, this method may further include the following optional steps. This step first verifies the audiovisual synchronization of the generated final animation and solidifies the verified parameter set into a reusable dynamic oral cavity model to improve the efficiency of subsequent animation generation. Furthermore, this step also identifies preset key phonemes in the speech input and adaptively adjusts the lip movement amplitude of the corresponding mouth shape animation based on the real-time environmental signal-to-noise ratio, thereby enhancing the visual recognizability of key information in noisy environments without affecting the overall naturalness.

[0064] Step S7: Verify and solidify the model, while strengthening the information transmission capability under extreme environments.

[0065] This step validates the final result and generates a reusable optimization model. For example... Figure 4 As shown, it also includes a robustness enhancement strategy for extreme environments.

[0066] From the final, consistent virtual digital human animation output, a set of verification frames is extracted according to certain rules (e.g., random or based on keyframes), and the lip movement features of these verification frames are obtained. These lip movement features are processed by a lightweight convolutional neural network (e.g., using an architecture similar to MobileNet or SqueezeNet). This network has a simple structure, low computational cost, and can be used for rapid verification. The temporal correlation between this lip shape change sequence and the waveform of the original speech input is evaluated, and the matching degree is determined by calculating the temporal correlation coefficient between the two. If the matching degree is higher than a preset threshold (e.g., 0.9), the audiovisual synchronization is considered good. The set of parameters relied upon in the current animation generation process, including the personalized oral cavity model, DNN mapping function weights, Kalman filter parameters, etc., is solidified to form a preliminary dynamic oral cavity model, which can be used for subsequent rapid animation generation tasks for this specific character. If the matching degree is lower than the threshold, the lip shape parameters are iteratively fine-tuned to improve synchronization until the matching degree reaches the target.

[0067] Furthermore, this invention employs a robustness enhancement strategy for key information transmission in extreme environments, ensuring the visual clarity of critical instructions in noisy conditions. This strategy specifically includes: the system has a built-in emergency vocabulary database containing high-priority words, such as "evacuation," "danger," and "immediately." Upon receiving voice input, the system converts it into a phoneme sequence and matches it against the emergency vocabulary database to identify key phonemes in the instructions. Subsequently, the system calculates the signal-to-noise ratio (SNR) of the current environment based on synchronously acquired ambient audio. For the identified key phonemes, when generating their lip-sync animation, the lip opening and closing speed and shape deformation coefficient are multiplied by an adaptive gain factor k. This factor k is inversely proportional to the SNR, and its calculation formula can be k = 1 + alpha / (SNR + epsilon), where alpha is a preset positive real gain coefficient, for example, a value between 0.5 and 2.0; epsilon is a very small positive number to prevent the denominator from being zero, for example, 1e-6. This means that the lower the signal-to-noise ratio and the noisier the environment, the larger the k-value. Therefore, without affecting the overall naturalness, the lip movements of key commands can be appropriately exaggerated, effectively enhancing visual clarity and helping the receiver understand key commands through visual aids even when auditory information is impaired. Finally, the parameters of this robust enhancement strategy are weighted and fused with the preliminary dynamic oral cavity model to obtain an optimized dynamic oral cavity model.

[0068] Example 2

[0069] This embodiment provides a virtual digital speech broadcast optimization device. This device can be a hardware entity such as a computer or server, and its hardware structure may include a processor, memory, and communication interface. The memory stores one or more computer programs, which can be executed by the processor to implement the virtual digital speech broadcast optimization method described in Embodiment 1 of this invention. Specifically, when the processor executes the program, it implements the following functional modules:

[0070] The data processing module is used to acquire facial image data of a specific person and synchronized speech signals, process the facial image data to extract preliminary facial features, and combine them with lip movement synchronization information extracted from the speech signals to fuse them into a final facial feature vector.

[0071] The structure generation module is used to generate a personalized description of the internal oral structure that reflects details such as the tooth arrangement and tongue shape of the specific person based on the final facial feature vector.

[0072] The trajectory generation module is used to acquire the speech signal to be played, analyze the speech signal to generate a preliminary driving parameter sequence, and then, in combination with the constraints of the personalized oral cavity internal structure description, optimize and map the preliminary driving parameter sequence into oral movement trajectory data that is coordinated with the personalized oral cavity internal structure description.

[0073] An animation synthesis module is used to generate an oral cavity animation containing detailed internal dynamics based on the oral movement trajectory data, and synthesize it with the overall facial animation data to generate a preliminary virtual digital human animation.

[0074] The feedback optimization module is used to extract facial key point sequences from the initial virtual digital human animation and the real reference video of the specific person, calculate the dynamic time warping (DTW) distance between the two sequences, thereby obtaining a quantified identity consistency index. When the identity consistency index exceeds a preset range, the index is used as the optimization target to retrospectively update the personalized oral cavity internal structure description, and the subsequent steps are repeated until the index converges to the preset range to output the final virtual digital human animation.

[0075] In this embodiment, the above-mentioned functional modules can be integrated into a single software application, stored in memory, and uniformly called and executed by the processor.

[0076] Example 3

[0077] This embodiment describes a specific application of the method of the present invention in a particular scenario: broadcasting public safety instructions with high timeliness and accuracy requirements using a virtual digital human. In this scenario, the broadcasting environment is a large transportation hub, where the ambient audio signal contains a significant amount of non-stationary background noise. The goal of the broadcasting task is to generate a virtual digital human that closely resembles a publicly recognized security spokesperson and ensures that the key instructions broadcast can be clearly received even in noisy environments.

[0078] First, because the speaker has unique dental alignment features, using existing methods based on generic oral templates would result in lip-sync animations that visually differ significantly from the speaker, reducing the broadcast's authority and credibility. Therefore, this embodiment employs steps S1 and S2. High-resolution facial video data of the speaker is acquired, their final facial feature vector is extracted, and this vector is used as a conditional input into a 3D conditional generative adversarial network (3D cGAN). The network generates a personalized description of the speaker's internal oral structure based on these external features, incorporating details of their unique tooth morphology and alignment. This step is fundamental to achieving highly consistent animation and solves the problem of existing technologies being unable to handle non-standard oral features in target individuals.

[0079] Next, the voice command to be broadcast is an emergency evacuation notice, containing key information such as "immediately" and "danger zone." During step S3, the system not only processes this voice command but also simultaneously analyzes the environmental audio collected from the transportation hub, calculating that the current environment's signal-to-noise ratio (SNR) is at a low level. To avoid environmental noise interfering with the mouth movement trajectory data and causing unnatural jitter in the animation, the system uses a Kalman filter to process the coordinated temporal structure data. By dynamically adjusting the filter parameters using the environmental noise spectrum as input, the movement trajectory is effectively smoothed, ensuring that the virtual digital human's mouth movements remain stable and smooth even when broadcasting in a simulated noisy environment.

[0080] Finally, to ensure the effective transmission of key instructions, this embodiment applies the robustness enhancement strategy for key information transmission from step S7. The system's built-in emergency vocabulary database identifies key phoneme sequences such as "immediately" and "danger zone" in the speech. Given the low signal-to-noise ratio at the scene, the system calculates an adaptive gain factor k greater than 1 and applies this factor to the lip-sync animation generation parameters corresponding to these key phonemes. Specifically, when these syllables are pronounced, the opening and closing speed and shape deformation coefficient of the lips are moderately increased. This processing visually enhances the lip movements of key information, allowing viewers to understand the core content of the instructions even when auditory information is impaired, by observing lip movements. Through the above steps, the final generated virtual digital human broadcast animation not only closely resembles the specific speaker in appearance and behavior, but also ensures the clarity and reliability of information transmission in harsh acoustic environments.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A virtual digital population broadcasting optimization method, characterized in that, Includes the following steps: Step 1: Acquire facial image data of a specific person and the corresponding voice signal. Process the facial image data to extract preliminary facial features, and combine them with the lip movement synchronization information extracted from the voice signal to form a final facial feature vector. Step 2: Based on the final facial feature vector, generate a personalized description of the oral cavity structure that reflects the tooth arrangement, tooth shape, tongue size and shape of the specific person. Step 3: Acquire the speech signal to be broadcast, analyze the speech signal to generate a preliminary driving parameter sequence, and then combine it with the constraints of the personalized oral cavity internal structure description to optimize and map the preliminary driving parameter sequence into mouth movement trajectory data that is consistent with the personalized oral cavity internal structure description. Step four: Based on the oral movement trajectory data, generate an oral cavity animation containing detailed internal dynamics, and synthesize it with the overall facial animation data to generate a preliminary virtual digital human animation; Step 5: By extracting facial key point sequences from the preliminary virtual digital human animation and the real reference video of the specific person, respectively, and calculating the dynamic time warping (DTW) distance between the two sequences, a quantitative identity consistency index is obtained. Step six: If the identity consistency index exceeds the preset range, the index is used as the optimization target, the personalized oral cavity internal structure description is updated retrospectively, and steps three to five are re-executed based on the updated personalized oral cavity internal structure description to obtain the identity consistency index again, until the re-obtained identity consistency index converges to the preset range, so as to output the final virtual digital human animation.

2. The method according to claim 1, characterized in that, In step one, extracting the lip movement synchronization information specifically includes: analyzing the energy envelope and phoneme timing of the speech signal, aligning and comparing them with the motion trajectories of key lip points in the image sequence corresponding to the facial image data, calculating the matching degree between lip movement and speech, and generating a lip movement enhancement vector based on the matching degree as the lip movement synchronization information.

3. The method according to claim 1, characterized in that, Step two, generating the personalized oral cavity internal structure description, specifically includes: The final facial feature vector is used as a conditional input and fed into a generator of a 3D conditional generative adversarial network pre-trained on a 3D oral cavity scan dataset to generate a 3D mesh model of the oral cavity containing details of tooth arrangement, tooth morphology, and tongue size and shape. Skeletal support vectors are extracted from the regions representing the maxilla and mandible in the three-dimensional mesh model inside the oral cavity, and these skeletal support vectors are fused with the three-dimensional mesh model inside the oral cavity to construct an enhanced oral cavity model. The structural stability of the enhanced oral cavity model is evaluated. If its deformation under simulated motion is lower than a preset threshold, then the enhanced oral cavity model is determined to be the personalized oral cavity internal structure description.

4. The method according to claim 1 or 3, characterized in that, Step three specifically includes: The rhythmic features and phoneme sequence of the speech signal to be broadcast are obtained, and a recurrent neural network is used to model the rhythmic features and phoneme sequence to generate a preliminary driving parameter sequence. The personalized oral cavity internal structure description is used as constraint information, and the preliminary driving parameter sequence is optimized through a parameter vector weighted adjustment module to determine the enhanced driving sequence; The enhanced driving sequence is used to optimize the mapping relationship between phonemes and corresponding lip shape parameters during the phoneme dynamic modeling process, and a speech driving parameter sequence is generated. Furthermore, based on the speech-driven parameter sequence and in conjunction with the constraints of the personalized oral cavity internal structure description, the speech-driven parameter sequence is mapped into oral movement trajectory data that is coordinated with the personalized oral cavity internal structure description.

5. The method according to claim 1, characterized in that, It also includes applying a robustness enhancement strategy for delivering critical information in extreme environments before outputting the final virtual digital human animation, the strategy including: It has a built-in emergency vocabulary database containing high-priority words, and converts the input voice signal to be broadcast into a phoneme sequence and matches it with the emergency vocabulary database to identify key phonemes in the instructions; The signal-to-noise ratio (SNR) of the current environment is calculated based on the environmental audio collected synchronously with the voice signal to be broadcast. For the identified key phonemes, when generating their lip-shape animation, their lip opening and closing speed and shape deformation coefficient are multiplied by an adaptive gain factor k, wherein the value of the adaptive gain factor k is inversely proportional to the signal-to-noise ratio.

6. The method according to claim 5, characterized in that, The adaptive gain factor k is calculated as follows: k = 1 + alpha / (SNR + epsilon), where alpha is a preset positive real gain coefficient, SNR is the signal-to-noise ratio, and epsilon is a very small positive number to prevent the denominator from being zero.

7. The method according to claim 1, characterized in that, The step of generating an oral movement trajectory that is coordinated with the personalized oral cavity internal structure description specifically includes: The temporal information corresponding to key phonemes is extracted from the speech driving parameter sequence, and Fourier transform is used to convert it into frequency features to obtain the temporal frequency distribution. The quantified coordination value is obtained by calculating the inner product of each frequency point vector in the temporal frequency distribution and the oral key point vector in the personalized oral internal structure description. A preset deep neural network mapping function is then adjusted according to the coordination value to output the coordinated temporal structure data. Noise components are extracted from synchronously acquired ambient audio signals to generate an ambient noise spectrum. An interference information represented by the ambient noise spectrum is then fused with the coordinated temporal structure data using a Kalman filter to determine a stationarity enhancement trajectory. The mouth movement trajectory data is then generated based on the stationarity enhancement trajectory.

8. The method according to claim 1, characterized in that, In generating the initial virtual digital human animation, an emotion fluctuation avoidance mechanism is also integrated, which includes: Real-time detection of three types of prosodic features in the input speech signal: pitch, speech rate, and energy, in order to identify prosodic feature peaks triggered by emotions; When generating the animation frame sequence, a smoothing filter is applied to the mouth movement trajectory at the time point corresponding to the prosodic feature peak to suppress mouth shape changes caused by emotional fluctuations.

9. The method according to claim 1, characterized in that, Before outputting the final, consistent virtual digital human animation, an information verification mechanism is also applied, which includes: The phoneme sequence corresponding to the final identity-consistent virtual digital human animation output is compared with the original phoneme sequence of the speech signal to be broadcast. If a mismatched or missing phoneme is found, the animation at that location will be automatically flagged and a correction process will be triggered.

10. A virtual digital population broadcast optimization device, used to perform the method as described in any one of claims 1-9, characterized in that, include: The data processing module is used to acquire facial image data of a specific person and synchronized speech signals, process the facial image data to extract preliminary facial features, and combine them with lip movement synchronization information extracted from the speech signals to fuse them into a final facial feature vector. The structure generation module is used to generate a personalized oral cavity internal structure description that reflects the tooth arrangement, tooth shape, tongue size and shape of the specific person based on the final facial feature vector. The trajectory generation module is used to acquire the speech signal to be played, analyze the speech signal to generate a preliminary driving parameter sequence, and then, in combination with the constraints of the personalized oral cavity internal structure description, optimize and map the preliminary driving parameter sequence into oral movement trajectory data that is coordinated with the personalized oral cavity internal structure description. An animation synthesis module is used to generate an oral cavity animation containing detailed internal dynamics based on the oral movement trajectory data, and synthesize it with the overall facial animation data to generate a preliminary virtual digital human animation. The feedback optimization module is used to extract facial key point sequences from the initial virtual digital human animation and the real reference video of the specific person, calculate the dynamic time warping (DTW) distance between the two sequences, thereby obtaining a quantified identity consistency index. When the identity consistency index exceeds a preset range, the index is used as the optimization target to retrospectively update the personalized oral cavity internal structure description, and trigger the trajectory generation module to regenerate mouth movement trajectory data based on the updated personalized oral cavity internal structure description. This triggers the animation synthesis module to regenerate the initial virtual digital human animation and recalculate the identity consistency index until the recalculated identity consistency index converges to the preset range, so as to output the final virtual digital human animation.

Images