Method for driving digital population type by voice

Abstract

The invention relates to the field of digital people, and discloses a method for driving a digital population pattern by voice, which comprises the following steps: receiving a voice signal and extracting a time-frequency feature, generating a personalized voiceprint feature based on the voice signal, generating a digital population pattern synchronous with the voice signal by using nonlinear dynamic modeling, and generating a digital population pattern synchronous with the voice signal. And precise synchronization of the mouth shape and the voice signal is ensured through a time sequence modeling method. Furthermore, facial expression generation is physically constrained through a Hamiltonian system, and it is ensured that the generated facial action conforms to the biomechanical law. Finally, a digital population type video is generated and output according to a synchronization result, and personalized, high-precision synchronous and natural facial expressions are achieved. The method is suitable for the fields of medical science popularization, virtual assistants and the like, and can improve the video generation efficiency and enhance the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of digital humans, and in particular to a method for driving digital human figures with voice. Background Art

[0002] Traditional methods for generating digital population models usually rely on manual recording and manual design of lip shapes and facial expressions. This process is complex and time-consuming, and the generated digital population models often cannot accurately reflect the dynamic characteristics of speech signals. These traditional methods make it difficult to achieve precise synchronization between speech and facial movements. Especially in personalized scenarios, the generation of lip shapes and facial expressions often lacks naturalness and realism. Existing automated generation methods mostly rely on general models. Although these models can handle basic lip shape generation tasks, they are difficult to personalize according to the speech characteristics of different users, resulting in the generated digital population models being unable to perfectly adapt to the pronunciation habits and speaking speed of different users, showing unnatural or uncoordinated visual effects.

[0003] In addition, most existing technologies do not fully consider the complex dynamic relationship between lip shape and facial expression. The correspondence between speech signals and facial expressions is nonlinear, and the movement of facial muscles has a high degree of biological regularity. In traditional methods, the lip shape generation process often ignores subtle changes in speech signals and fails to consider the emotional fluctuations of speech and its impact on facial movements, resulting in the lack of emotional expression and dynamic changes in the generated digital mouth shape.

[0004] Furthermore, existing synchronization methods generally rely on static algorithm models and are unable to process the temporal features in speech signals in real time and efficiently, resulting in a less than smooth dynamic synchronization process between speech signals and facial expressions. Even in some efficient deep learning models, although they can process some temporal features of speech signals, they are still insufficient in capturing long-term dependencies and emotional features, making it difficult for the generated video content to fully present the emotional fluctuations and detailed changes in speech signals.

[0005] Therefore, the existing technology has significant deficiencies in terms of personalized lip shape generation, natural synchronization of speech and facial expressions, and dynamic changes of facial expressions. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a method for driving a digital population type with voice, which solves the problems in the prior art of inaccurate synchronization between the generation of a digital population type and a voice signal, difficulty in personalized adjustment, and unnatural facial expressions.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for voice-driven digital population type, comprising the following steps: Receiving a speech signal, and preprocessing the speech signal to extract time-frequency features of the speech signal; Generate a personalized voiceprint feature based on the time-frequency feature of the voice signal, wherein the voiceprint feature is related to the personalized voice feature of the doctor; Generate a digital mouth shape synchronized with the speech signal using nonlinear dynamics modeling, wherein the nonlinear dynamics modeling describes the nonlinear relationship between the speech signal and the mouth shape based on a chaotic dynamics model; Synchronizing the speech signal with the digital mouth shape through a timing modeling method to ensure that the mouth shape is aligned with the speech content; According to the generated synchronization result, a final digital population-type video is generated and output.

[0008] Preferably, the time-frequency features of the speech signal are extracted by short-time Fourier transform, and the short-time Fourier transform is based on framing of the speech signal to calculate the spectrum information of each frame to obtain a joint representation of time and frequency.

[0009] Preferably, the generation of the personalized voiceprint features is achieved through the joint implementation of a convolutional neural network and a long short-term memory network, wherein the convolutional neural network is used to extract local features of the speech signal, and the long short-term memory network is used to capture the temporal relationship in the speech signal to obtain the personalized voice features of the doctor.

[0010] Preferably, the step of using nonlinear dynamics modeling to generate a digital population shape synchronized with the speech signal comprises: Based on the time-frequency characteristics of the speech signal, a nonlinear dynamic model is constructed, wherein changes in the speech signal are associated with facial movements of the digital population through a state equation of the dynamic system; The state equation is as follows:

[0011] in, is the state vector of the system, representing the state of the digital population type; is the input vector of the speech signal, including the time-frequency features extracted from the speech signal; are the parameters of the model, is a function that describes the change of system state; The nonlinear relationship between the speech signal and the digital population profile is modeled by the state equation, and a digital population profile synchronized with the speech signal is generated.

[0012] Preferably, the step of using nonlinear dynamics modeling to generate a digital population synchronized with the speech signal further comprises: The stability analysis was performed on the modeling system using the Lyapunov index, which was calculated using the following formula:

[0013] in, Representation System The disturbance term at time, represents the disturbance term at the initial time of the system, is the Lyapunov exponent; By calculating the Lyapunov exponent, the stability of the digital mouth shape generation process is ensured to avoid unnatural mouth shape generation.

[0014] Preferably, the step of using nonlinear dynamics modeling to generate a digital population synchronized with the speech signal further comprises: The Hamiltonian system is used to constrain the digital population generation process. The motion equation of the Hamiltonian system is:

[0015] in, is the Hamiltonian; is the displacement coordinate of the digital population face, indicating the physical position of each part of the face in the digital population; is momentum, which indicates the movement state of facial muscles; is the mass of facial muscles, indicating the movement inertia of facial muscles; is the potential energy function, which represents the potential energy of facial muscles; and denote the derivatives of Hamiltonian with respect to position and momentum respectively; Through the Hamiltonian system constraints, it is ensured that the digital population generation follows the laws of physics and the movement of facial muscles matches the changes in the speech signal.

[0016] Preferably, the step of performing time synchronization on the voice signal and the digital population model by using a timing modeling method comprises: The speech signal and the digital mouth shape are time-aligned using a temporal modeling method, and the temporal relationship between the speech signal and the mouth shape is learned through a bidirectional long short-term memory network. The time-frequency features and digital mouth shape features of the speech signal are input and processed through a bidirectional long short-term memory network to obtain a feature output containing forward and backward timing information, adjust the time delay of mouth shape generation, and ensure accurate synchronization between speech and mouth shape.

[0017] Preferably, the step of generating a final digital population-type video according to the generated synchronization result and outputting the video comprises: combining the digital population pattern synchronized by the time series modeling method with the speech signal; According to the synchronization result, adjusting the facial expression of the digital population to match the emotional information of the voice signal; Based on generating each frame image of the digital population, the image includes facial movements and lip dynamics; Combine the generated image sequences into a video file in chronological order, where each frame of the image represents the facial expression and lip shape of the digital population at a specific moment; The video file is output so that it can be viewed and used by users.

[0018] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above method is implemented.

[0019] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0020] The present invention provides a method for driving digital population by voice, which has the following beneficial effects: The present invention realizes accurate synchronization between speech signals and digital mouth shapes through nonlinear dynamic modeling and time series modeling methods. The pronunciation of each syllable can be accurately mapped to the movement of facial muscles, ensuring that the generated video is highly consistent with the speech content in terms of lip shape changes.

[0021] By extracting the personalized voiceprint features of doctors, the digital population generated by the present invention is not only synchronized with the voice, but also able to reflect the unique voice features of doctors, such as pitch, speaking speed and timbre, etc. This enables the generated video to more realistically restore the pronunciation style and emotional expression of doctors.

[0022] The present invention introduces the Hamiltonian system to physically constrain the generation of facial expressions, ensuring that the movement of facial muscles conforms to the laws of biomechanics. Facial expressions are not only synchronized with the emotional information of voice signals, but also present a more natural and realistic visual effect.

[0023] Through the technology of the present invention, the generation process of digital population video is optimized, the synchronization of voice signals and lip shape is greatly improved, and the video output is fast, avoiding the complicated manual production process in the traditional method and significantly improving the efficiency.

[0024] The digital population video generation technology of the present invention can be widely used in many fields such as medical popularization, virtual assistants, entertainment and education. It can not only provide personalized virtual character performance, but also enhance the audience's immersion and interactivity, and has broad market potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the method flow of the present invention; Figure 2 It is a schematic diagram of the structure of the device of the present invention; Figure 3 It is a schematic diagram of the computer device structure of the present invention.

[0026] Among them, 100, speech signal input module; 200, speech signal processing module; 300, voiceprint feature extraction module; 400, lip shape generation module; 500, synchronization module; 600, output module; 40, computer equipment; 41, processor; 42, memory; 43, storage medium. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Please see attached Figure 1 The present invention provides a method for voice-driven digital population type, which aims to generate a digital population type video synchronized with the doctor's voice according to the doctor's voice signal. The method combines a variety of technical means to ensure that the generated video lip shape is highly consistent with the voice content and can reflect the doctor's personalized voice characteristics. The core of this implementation includes feature extraction of voice signals, personalized voiceprint generation, nonlinear dynamic modeling, time series modeling synchronization, and the generation and output of the final digital population type video.

[0029] like Figure 1 As shown, the method for driving the digital population by voice may include the following steps: S1, receiving a speech signal, preprocessing the speech signal, and extracting the time-frequency features of the speech signal; S2, Generate personalized voiceprint features based on the time-frequency features of the speech signal; S3, using nonlinear dynamics modeling to generate digital population patterns synchronized with speech signals; S4, time synchronization between the speech signal and the digital population type through a time series modeling method; S5. Generate a final digital population video according to the generated synchronization result, and output the video.

[0030] The method steps and technical implementation of the present invention are described in detail below.

[0031] For step S1, in this embodiment, by receiving and processing the voice signal, the necessary input is provided for the subsequent personalized voiceprint feature extraction and lip shape generation. The specific operation process is as follows: In this step, the doctor's voice signal is first received through a suitable voice input device (such as a microphone, voice acquisition card, etc.). The received voice signal is usually a time domain signal, which contains the doctor's pronunciation information. The quality of the voice signal directly affects the subsequent processing results, so the signal needs to be preprocessed first.

[0032] The key to speech signal preprocessing is to remove noise and eliminate unnecessary signal interference. In order to improve the clarity of the signal, a bandpass filter or adaptive filtering method can be used to remove noise. For example, a Wiener filter is used to smooth the speech signal to reduce the impact of background noise and other irrelevant frequencies. In addition, the amplitude of the speech signal can be adjusted through a normalization process so that the dynamic range of the signal is suitable for the subsequent feature extraction process.

[0033] After completing the preprocessing of the speech signal, the time-frequency feature extraction is performed. Since the speech signal is essentially non-stationary, that is, its spectrum changes with time, it needs to be processed by time-frequency analysis technology. In the present invention, short-time Fourier transform (STFT) and wavelet transform are the main methods for extracting the time-frequency features of the speech signal.

[0034] Specifically, the short-time Fourier transform (STFT) is used to frame the speech signal and calculate the spectrum information of each frame to obtain a joint representation of time and frequency. The specific operation can be achieved by the following formula:

[0035] in, is the input speech signal, is the window function, It is the time-frequency representation after short-time Fourier transform. Through framing and Fourier transform, the spectrum information of each frame can be obtained, thereby capturing the changes of the signal in time and frequency.

[0036] In addition, wavelet transform, as another time-frequency analysis method, can capture the local characteristics of the signal more accurately. Wavelet transform can not only capture low-frequency information through multi-scale analysis, but also effectively extract high-frequency components. In this embodiment, wavelet transform is used to extract time-frequency features, and the specific formula is as follows:

[0037] in, is the wavelet basis function, and are the scale and translation parameters, is the wavelet transform coefficient. Through this method, multi-scale analysis of speech signals can be obtained, making the time-frequency feature extraction more refined.

[0038] Through STFT and wavelet transform, the extracted time-frequency features provide the basis for the generation of personalized voiceprint features in the subsequent steps. These time-frequency features capture the spectral information of the speech signal and can effectively reflect the personalized features of the speech, such as pitch and speaking speed.

[0039] The time-frequency features extracted in this step have the following advantages: first, they can describe the changes in time and frequency of speech signals and provide rich speech information; second, the time-frequency features can help accurately match the pronunciation habits of doctors in the generation of medical popular science videos and provide accurate input for the generation of personalized digital population types.

[0040] In summary, step S1 receives the voice signal, performs preprocessing, and obtains the time-frequency features of the voice using the time-frequency feature extraction method, laying the foundation for the subsequent generation of personalized voiceprint features and digital population type.

[0041] As for step S2, in this embodiment, step S2 generates voiceprint features related to the personalized features of the doctor's voice based on the time-frequency features of the voice signal extracted in step S1. By extracting the voiceprint features, the personalized pronunciation features of each doctor can be effectively captured to ensure that the generated digital population can be highly matched with the doctor's voice features.

[0042] In this embodiment, the time-frequency features of the speech signal are the spectrum information extracted from the doctor's speech signal. These time-frequency features can reflect the personalized features of the speech, such as pitch, timbre, and speaking speed. Based on these time-frequency features, the convolutional neural network (CNN) and long short-term memory network (LSTM) in the deep learning method are used to extract voiceprint features.

[0043] First, convolutional neural networks (CNNs) are used to extract local features from time-frequency graphs. The time-frequency features of speech signals are usually spectrograms obtained through short-time Fourier transform or wavelet transform. Convolutional neural networks can process these spectrograms through their convolution layers to capture local feature patterns. The advantage of convolutional neural networks is that they can effectively extract local information from the spectrum, such as syllables, pitch, and tone changes, through convolution operations.

[0044] Secondly, the long short-term memory network (LSTM) is used to capture the temporal information of the voice signal, especially the dynamic changes of pitch and intonation. As a recursive neural network with long-term memory, LSTM can handle the dependencies with longer time spans in voice signals. Therefore, the LSTM network plays a key role in voiceprint feature extraction, and can associate the local features extracted from CNN with the time series to generate more accurate personalized voiceprint features.

[0045] The specific operation process is as follows: Time-frequency feature input: The time-frequency features of the speech signal obtained in step S1 are used as the input of CNN. These features can reflect the changes of the speech signal at different time points and frequencies, thereby helping the network capture the local features of the speech.

[0046] Convolutional layer processing: CNN performs convolution processing on the time-frequency graph through multiple convolutional layers to extract local features, such as syllables, timbre, etc. These local features are crucial for describing the personalized characteristics of the doctor's voice. The convolution process processes the speech features within each time window by sliding the convolution kernel to capture local changes.

[0047] Temporal information extraction: After extracting local features, these features are further processed using an LSTM network to capture the temporal dependencies in the speech signal. LSTM is able to process dynamic changes over long time spans in speech signals and generate feature vectors related to the personalization of the doctor’s speech.

[0048] The generated personalized voiceprint features represent the uniqueness of the doctor's voice, such as pitch, speaking speed, timbre, etc., and provide a basis for the subsequent generation of digital population profiles. These voiceprint features can not only ensure that the digital population profile is highly synchronized with the doctor's voice, but also reflect the personalized characteristics of the doctor's voice, making the generated digital population profile more personalized.

[0049] In this embodiment, by combining CNN and LSTM, we successfully achieved deep mining of the time-frequency features of the speech signal and generated voiceprint features that are closely related to the personalized speech features of the doctor. This process not only improves the accuracy of digital population generation, but also enhances the personalization of the generated results, making the generated video content more realistic and consistent with the doctor's speech characteristics.

[0050] For step S3, in this embodiment, a digital mouth shape synchronized with the speech signal is generated through nonlinear dynamics modeling. In this step, the chaotic dynamics model is used to capture the nonlinear relationship between the speech signal and the digital mouth shape, and ensure that the generated mouth shape is highly synchronized with the speech signal.

[0051] In the process of generating digital mouth shape, the relationship between speech signal and mouth shape is complex and nonlinear. The change of speech signal will directly affect the movement of facial muscles, and this influence is nonlinear and time-varying. In order to accurately describe this complex relationship, this embodiment adopts a chaotic dynamics model, which can effectively capture the nonlinear characteristics existing in the system.

[0052] First, based on the time-frequency characteristics of the speech signal (e.g., the spectrogram obtained by short-time Fourier transform or wavelet transform), a state equation of a dynamic system is established. This equation describes how the speech signal input affects the facial muscle dynamics of the digital population.

[0053] The state equation of the dynamic system is expressed as:

[0054] in: The state vector representing the digital population describes the movement of facial muscles, such as the dynamic changes of lips, tongue and jaw; The input signal contains the time-frequency features of the speech signal. Specifically, these features include the frequency, amplitude, and syllables of the audio. It is a set of system parameters that controls the response characteristics of facial muscles, such as muscle tension, movement characteristics of facial bones, etc. It is a nonlinear function that describes the change of system state, which links the speech signal with the dynamic response of facial muscles.

[0055] The model is based on the chaotic dynamics method, through the nonlinear function Accurately simulate the nonlinear mapping relationship between speech and mouth shape, ensuring that the mouth shape at each time point is consistent with the instantaneous characteristics of the speech signal.

[0056] In order to ensure that the generated digital mouth shape is physically stable and reasonable, this embodiment introduces the Lyapunov index to analyze the stability of the system. The Lyapunov index is an important indicator to measure the sensitivity of a dynamic system to initial conditions. By calculating the Lyapunov index, the stability of the model when generating the mouth shape can be evaluated to avoid unnatural mouth shape changes in the system.

[0057] In this embodiment, the calculation formula of the Lyapunov exponent is as follows:

[0058] in: Indicates that the system is at time The disturbance term at each moment reflects the change in the state of facial muscles; is the system initial time The disturbance term of is the Lyapunov exponent, which indicates the stability of the system. , the system is unstable; if , the system tends to be stable.

[0059] In this embodiment, by calculating the Lyapunov exponent, the stability of the model can be evaluated, and the system parameters (i.e. ), to ensure that the generated mouth shape changes are smooth and natural, and to avoid unstable facial movements.

[0060] In order to ensure that the movement of facial muscles conforms to the laws of biomechanics, this embodiment further introduces a Hamiltonian system, through which physical constraints are imposed on the lip shape generation process. The Hamiltonian system can simulate the movement of physical systems, and is particularly suitable for describing the behavior of particles in a force field. In this embodiment, the Hamiltonian system is used to describe the dynamic response of facial muscles, thereby ensuring that the generated digital mouth shape conforms to the laws of physics.

[0061] Specifically, the equations of the Hamiltonian system are as follows:

[0062] in: is the Hamiltonian, which represents the total energy of the system, including kinetic energy and potential energy; It is the displacement coordinates of the facial muscles of the digital population, indicating the physical positions of various parts of the face (such as mouth, eyes, chin, etc.); is momentum, which indicates the movement state of facial muscles; is the mass of facial muscles, describing the inertia of facial muscles; is the potential energy function, which represents the potential energy of facial muscles and is related to the geometric shape of the face and the tension of the muscles; and They represent the partial derivatives of Hamiltonian with respect to position and momentum, respectively, describing the dynamic changes of facial muscles.

[0063] Through the Hamiltonian system, the movement of facial muscles is not only driven by speech signals, but also follows the laws of physics, thus ensuring that the generated lip shape is biologically reasonable.

[0064] Through the above-mentioned nonlinear dynamic modeling, Lyapunov exponent stability analysis and Hamiltonian system constraints, the speech signal is combined with the facial dynamic changes of the digital population. The generated digital population is not only highly synchronized with the speech signal, but also meets the requirements of biomechanics in terms of movement, avoiding unnatural lip shape and facial expressions.

[0065] Through this model, each syllable of the speech signal, the change in speech speed, etc. can be accurately converted into facial movements, ensuring that the generation of digital mouth shapes is both accurate and natural. In particular, the introduction of Lyapunov exponents and Hamiltonian system constraints greatly enhances the stability and physical rationality of the generation process, ensuring that the generated mouth shapes and facial expressions not only conform to the rhythm and emotional changes of the speech content, but also conform to the biomechanical characteristics, making the final generated video more realistic and credible.

[0066] For step S4, in this embodiment, the time series modeling method is used to ensure the accurate synchronization of the voice signal and the digital mouth shape in time. In the process of generating the digital mouth shape, the change of the mouth shape must be highly matched with the rhythm, syllables and dynamic characteristics of the voice signal to ensure that the final generated video content is natural and conforms to the actual pronunciation rules.

[0067] To this end, the present invention uses a bidirectional long short-term memory network (Bi-LSTM) for time series modeling. Bi-LSTM is a deep learning model with powerful time series learning capabilities. It can capture the long-term dependency between speech signals and lip shape changes, ensuring that the lip shape and speech signals are accurately aligned in time.

[0068] In this embodiment, the time-frequency features of the speech signal extracted from steps S1 and S2 are first used as the input sequence of the Bi-LSTM model. These time-frequency features can fully reflect the changes in the time and frequency of the speech signal, providing key data support for time series modeling.

[0069] The Bi-LSTM model can capture the temporal information in the speech signal more comprehensively through the learning mechanism of forward propagation and backward propagation. By forward and backward propagation of the input speech signal features, Bi-LSTM can process information in two time directions respectively, thereby capturing the past and future features of the speech at the same time. This feature enables Bi-LSTM to model the lip shape more accurately and synchronize the lip shape with the speech signal precisely.

[0070] In the specific implementation process, the state update formula of the Bi-LSTM network is:

[0071] in: is the output at the current moment, indicating the generated lip state; and are the states at the previous and next moments, respectively, representing the forward and backward information of the speech signal; is the voice signal feature input at the current moment; is the weight matrix, responsible for mapping the input information to the output space; is a bias term used to adjust the offset of the model output.

[0072] Through this formula, the Bi-LSTM network can capture the dynamic changes at each moment in the time series, thereby generating lip shapes synchronized with the speech signal.

[0073] In addition, the Bi-LSTM model can automatically adjust the duration, intensity and transition of the mouth shape during the timing synchronization process to make it highly matched with the syllables, pitch, pauses and other features of the speech signal. The model continuously learns the rhythm, speech speed and tone changes in the speech signal, and dynamically adjusts the changes in the mouth shape of each frame to ensure that the generated digital mouth shape has a natural and accurate synchronization effect.

[0074] Through time series modeling, Bi-LSTM can not only handle short-term speech changes, but also capture the dynamic characteristics of speech over a long time span. In this way, the generation process of the digital mouth shape is not only synchronized with the current syllable, but also able to adapt to long-term changes in speech characteristics, thereby ensuring the coherence and fluency of the mouth shape changes.

[0075] Through step S4, the synchronization between the digital mouth shape change and the speech signal is significantly enhanced. The generation of each frame of mouth shape is dynamically adjusted based on the timing characteristics of the speech signal, so that the final generated digital mouth shape has a high degree of naturalness and consistency, and can accurately reflect the rhythm, emotion and pronunciation characteristics of the speech.

[0076] This embodiment ensures a high degree of synchronization between the lip shape and the voice signal by introducing the Bi-LSTM network for timing modeling. The bidirectional information transmission mechanism of the Bi-LSTM network enables the lip shape generation to not only take into account the voice features of the current moment, but also make full use of the voice information of the previous and next moments for prediction, thereby improving the accuracy of timing synchronization. This method effectively eliminates the unnatural feeling caused by the asynchrony between the voice signal and the lip shape changes in the traditional method, while improving the fluency and realism of the digital mouth shape generation.

[0077] Through this step, the synchronization process between the voice signal and the lip shape is precisely aligned, and the generated digital mouth shape is consistent with the doctor's voice signal, which is not only visually realistic and vivid, but also able to reflect the detailed changes in the voice. This method provides an efficient and natural digital mouth shape generation technology for applications such as medical popular science videos and virtual doctors, improving user experience and enhancing the communication effect of video content.

[0078] For step S5, in this embodiment, according to the synchronization result previously generated by the time series modeling method, a digital population video is finally generated, and the video is output for the user to use. The technology involved in this step mainly includes converting the facial movements and lip shapes of the digital population after synchronization with the voice signal into each frame of image, and combining them into a complete video file, ensuring that the video content is natural and smooth, and highly consistent with the lip shape, rhythm and emotional changes of the voice signal.

[0079] In this embodiment, first, according to the synchronization result generated by the time series modeling method in step S4, the facial expression of the digital population is adjusted to match the emotional information of the voice signal. In this way, the digital population is not only synchronized with the voice signal in terms of lip shape, but also synchronized with the emotional changes of the voice in terms of expression.

[0080] The facial action generation process for each frame is as follows: Facial movement and lip shape generation: Based on the synchronization results, the movement of facial muscles is generated through the model. These facial muscle movements include dynamic changes in lips, chin, eyes, etc. Each frame of the image reflects the lip shape and facial expression of the digital population at a certain moment.

[0081] Mathematical modeling of the generation process: When generating each frame of an image, a facial model in computer graphics (such as 3D facial modeling) and muscle drive technology (such as a facial animation drive model) can be used to model the movement of facial parts. Based on the nonlinear dynamics model provided in step S3, the movement state of the facial muscles is calculated and driven by parameterized equations (such as a facial muscle model).

[0082] The generation of each frame of digital population image takes into account the following factors: Dynamic response of facial muscles: The movements and tension of facial muscles are dynamically adjusted according to the syllables and speech rate in the speech signal. The specific facial movement model can be described by the motion equations of facial muscles and adjusted accordingly based on the generated timing synchronization results.

[0083] Facial expression matching: In addition to lip syncing, facial expressions (such as smiles, frowns, etc.) are also matched based on the emotional information of the voice signal. Through the sentiment analysis algorithm, the emotional features in the voice signal (such as happiness, sadness, surprise, etc.) are identified, and then these emotional features are mapped to the facial expressions of the digital population.

[0084] Each frame of the generated image is arranged in time sequence to form a series of continuous image frames. The specific process includes: Image synthesis: The digital population facial image generated in each frame is synthesized with the background scene to generate an image that contains complete facial expressions and background. The synthesized image maintains the synchronization and emotional consistency of the speech signal.

[0085] Video synthesis: Combine continuous image frames into a final video file through video synthesis technology (such as encoding technology). The frame rate, resolution and format of the video can be adjusted according to specific needs. Common video formats include MP4, AVI, MOV, etc.

[0086] The generated digital population-type video is not only highly synchronized, but also can truly reflect the syllables, emotions and speech speed changes in the voice, making the video content natural and highly interactive and immersive.

[0087] Finally, the generated video files will be output in a standard video encoding format, allowing users to view, share and use the generated video content. The output formats of the video can include but are not limited to MP4 format, AVI format, etc., ensuring a wide range of compatibility and application scenarios.

[0088] Through step S5 in this embodiment, the generated digital population video can be applied in many fields such as medical popularization, virtual assistants, and entertainment. The digital population in the video is highly synchronized with the doctor's voice signal. The lip shape and facial expression not only conform to the rhythm and emotional changes of the voice content, but also reflect the personalized voice characteristics of the doctor. The implementation of this technology makes the video content more vivid and real, and improves the user's acceptance and understanding of the video content.

[0089] In addition, in the process of generating digital population-type videos, the physical constraints of facial muscles and emotional mapping are fully considered, so that the generated videos are not only accurately synchronized in visual effects, but also able to convey the emotional information in the voice, thereby enhancing the interactivity and expressiveness of the video.

[0090] Through this step, the present invention can realize the efficient conversion from voice signal to digital population type video, and provides a novel and personalized digital video generation method, which has important application value and broad market prospects.

[0091] In general, the present invention achieves a high degree of synchronization between digital population type and voice signal by receiving voice signal and extracting time-frequency features, combining personalized voiceprint feature generation with nonlinear dynamic modeling. This method adopts chaotic dynamic model and time series modeling technology, synchronizes voice signal with digital population type through bidirectional long short-term memory network (Bi-LSTM), and combines Hamiltonian system to physically constrain facial expression generation, ensuring that the generated digital population type fully matches the voice rhythm, emotion and speaking speed. Finally, by generating and synthesizing video files frame by frame, the generation of digital population type videos that are highly consistent with the voice content, natural and vivid, and have personalized characteristics is achieved. It is widely used in medical popular science, virtual assistants and other fields, and has important application value and market prospects.

[0092] The apparatus for driving a digital population using voice described below and the method for driving a digital population using voice described above may be referred to in correspondence with each other.

[0093] Please see attached Figure 2 The present invention also provides a device for voice-driven digital population, comprising: The voice signal input module 100 is used to receive a voice signal; The speech signal processing module 200 is used to pre-process the speech signal and extract the time-frequency features of the speech signal; A voiceprint feature extraction module 300 is used to generate personalized voiceprint features related to personalized features of the doctor's voice; A lip shape generation module 400 generates a digital mouth shape synchronized with the speech signal using nonlinear dynamics modeling based on the time-frequency features and the personalized voiceprint features; A synchronization module 500, configured to perform time synchronization between the speech signal and the digital population model by using a time series modeling method; The output module 600 is used to generate and output the final digital population video.

[0094] The device of this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0095] Please see attached Figure 3 The present invention further provides a computer device 40, comprising: a processor 41 and a memory 42, wherein the memory 42 stores a computer program executable by the processor, and when the computer program is executed by the processor, the above method is executed.

[0096] The present invention further provides a storage medium 43 on which a computer program is stored. When the computer program is run by the processor 41, the above method is executed.

[0097] Among them, the storage medium 43 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0098] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for voice-driven digital population profiling, characterized in that: The following steps are involved: Receiving a speech signal, and preprocessing the speech signal to extract time-frequency features of the speech signal; Generating personalized voiceprint features based on the time-frequency features of the voice signal; Generate a digital mouth shape synchronized with the speech signal using nonlinear dynamics modeling, wherein the nonlinear dynamics modeling describes the nonlinear relationship between the speech signal and the mouth shape based on a chaotic dynamics model; Synchronizing the speech signal with the digital mouth shape through a timing modeling method to ensure that the mouth shape is aligned with the speech content; According to the generated synchronization result, a final digital population-type video is generated and output.

2. The method for voice-driven digital population type according to claim 1, characterized in that: The time-frequency features of the speech signal are extracted by short-time Fourier transform, which calculates the spectrum information of each frame based on the frame processing of the speech signal to obtain a joint representation of time and frequency.

3. The method of voice-driven digital population type according to claim 1, characterized in that: The generation of the personalized voiceprint features is achieved through the joint implementation of a convolutional neural network and a long short-term memory network. The convolutional neural network is used to extract local features of speech signals, and the long short-term memory network is used to capture the temporal relationship in speech signals to obtain the personalized voice features of doctors.

4. The method for voice-driven digital population type according to claim 1, characterized in that: The step of using nonlinear dynamics modeling to generate a digital population synchronized with the speech signal comprises: Based on the time-frequency characteristics of the speech signal, a nonlinear dynamic model is constructed, wherein changes in the speech signal are associated with facial movements of the digital population through a state equation of the dynamic system; The state equation is as follows: ； in, is the state vector of the system, representing the state of the digital population type; is the input vector of the speech signal, including the time-frequency features extracted from the speech signal; are the parameters of the model, is a function that describes the change of system state; The nonlinear relationship between the speech signal and the digital population profile is modeled by the state equation, and a digital population profile synchronized with the speech signal is generated.

5. The method for voice-driven digital population type according to claim 4, characterized in that: The step of using nonlinear dynamics modeling to generate a digital population synchronized with the speech signal further comprises: The stability analysis was performed on the modeling system using the Lyapunov index, which was calculated using the following formula: ； in, Representation System The disturbance term at time, represents the disturbance term at the initial time of the system, is the Lyapunov exponent; By calculating the Lyapunov exponent, the stability of the digital mouth shape generation process is ensured to avoid unnatural mouth shape generation.

6. The method for voice-driven digital population modeling according to claim 5, characterized in that: The step of using nonlinear dynamics modeling to generate a digital population synchronized with the speech signal further comprises: The Hamiltonian system is used to constrain the digital population generation process. The motion equation of the Hamiltonian system is: ； in, is the Hamiltonian; is the displacement coordinate of the digital population face, indicating the physical position of each part of the face in the digital population; is momentum, which indicates the movement state of facial muscles; is the mass of facial muscles, indicating the movement inertia of facial muscles; is the potential energy function, which represents the potential energy of facial muscles; and denote the derivatives of Hamiltonian with respect to position and momentum respectively; Through the Hamiltonian system constraints, it is ensured that the digital population generation follows the laws of physics and the movement of facial muscles matches the changes in the speech signal.

7. The method of voice-driven digital population type according to claim 1, characterized in that: The step of performing time synchronization between the voice signal and the digital population model by using a time series modeling method comprises: The speech signal and the digital mouth shape are time-aligned using a temporal modeling method, and the temporal relationship between the speech signal and the mouth shape is learned through a bidirectional long short-term memory network. The time-frequency features and digital mouth shape features of the speech signal are input and processed through a bidirectional long short-term memory network to obtain a feature output containing forward and backward timing information, adjust the time delay of mouth shape generation, and ensure accurate synchronization between speech and mouth shape.

8. The method for voice-driven digital population modeling according to claim 1, characterized in that: The step of generating a final digital population video according to the generated synchronization result and outputting the video comprises: combining the digital population pattern synchronized by the time series modeling method with the speech signal; According to the synchronization result, adjusting the facial expression of the digital population to match the emotional information of the voice signal; Based on generating each frame image of the digital population, the image includes facial movements and lip dynamics; Combine the generated image sequences into a video file in chronological order, where each frame of the image represents the facial expression and lip shape of the digital population at a specific moment; The video file is output so that it can be viewed and used by users.

9. A device for voice-driven digital population type, used for executing the method for voice-driven digital population type as claimed in any one of claims 1 to 8, characterized in that: include: A voice signal input module, used for receiving a voice signal; A speech signal processing module, used for preprocessing the speech signal and extracting the time-frequency features of the speech signal; A voiceprint feature extraction module, used to generate personalized voiceprint features related to the personalized features of the doctor's voice; A lip shape generation module, based on the time-frequency features and the personalized voiceprint features, uses nonlinear dynamics modeling to generate a digital mouth shape synchronized with the speech signal; A synchronization module, used for performing time synchronization between the voice signal and the digital population model by using a time series modeling method; The output module is used to generate and output the final digital population video.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for voice-driven digital population modeling as described in any one of claims 1 to 8 is implemented.

Images