Digital population type and voice alignment method and device, electronic equipment and storage medium

The method uses machine learning models to synchronize digital human lip shape and speech, addressing the inefficiencies of manual alignment methods by enhancing automation and real-time performance, thus improving user interaction and reducing costs.

CN119068903APending Publication Date: 2024-12-03ZHEJIANG GEELY HLDG GRP CO LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411012022.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing methods for aligning digital human lip shape and speech rely heavily on manual production and preset rules, leading to uneven alignment and low automation, resulting in poor real-time performance and user experience.

Method used

A method utilizing machine learning models, including a speech synthesis model, lip shape generation model, and voice and lip shape alignment model, to synchronize digital human lip shape and speech in real-time, improving automation and efficiency.

Benefits of technology

Enhances the synchronization and timeliness of digital human speech and lip shape alignment, improving user interaction and reducing production costs while ensuring accurate and natural responses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068903A_ABST
    Figure CN119068903A_ABST
Patent Text Reader

Abstract

The invention provides a digital population type and voice alignment method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps of determining reply text information responding to input data according to the input data; converting the reply text information into voice information by adopting a preset voice synthesis model, and converting the reply text information or the voice information into mouth shape change information by adopting a preset mouth shape generation model; and aligning the voice information and the mouth shape change information on a time sequence by adopting a preset voice and mouth shape alignment model, and controlling the digital person to synchronously output the aligned voice information and mouth shape change information. According to the invention, the mouth shape driving of the digital human is realized based on the preset speech synthesis model, the preset mouth shape generation model, the preset speech and mouth shape alignment model and the like, and the synchronization of the speech and the mouth shape of the digital human can be ensured and the efficiency of the alignment of the digital human body and the speech can be improved while the automation degree of the alignment of the digital human body and the speech is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for aligning digital population type and voice. Background Art

[0002] Digital Human (Digital Human / Meta Human) is a digital image close to human image created by artificial intelligence technology and digital technology. With the rapid development of related technologies, the application scenarios of Digital Human are becoming more and more extensive, which greatly enriches people's experience in life, work and entertainment. Among them, when users interact with Digital Human by voice, the alignment effect between the Digital Human's mouth shape and its output voice has an important impact on the user's experience.

[0003] At present, the method of driving the alignment of digital human lip shape and speech mainly relies on preset rules and manual production by production staff. This method is highly dependent on the experience and skill level of the production staff, resulting in uneven alignment of digital human lip shape and speech. In addition, the process of achieving alignment of digital human lip shape and speech has a low degree of automation, resulting in poor real-time performance (i.e. efficiency) of the alignment process. These problems reduce the overall experience of users interacting with digital humans. Summary of the invention

[0004] The problem solved by the present invention is: how to improve the automation and efficiency of aligning digital population type with voice.

[0005] To solve the above problems, the present invention provides a method for aligning digital population type and voice, comprising:

[0006] Determining, based on the input data, reply text information in response to the input data;

[0007] Using a preset speech synthesis model to convert the reply text information into speech information, and using a preset lip shape generation model to convert the reply text information or the speech information into lip shape change information;

[0008] The voice information and the lip shape change information are aligned in time series using a preset voice and lip shape alignment model, and the digital human is controlled to synchronously output the aligned voice information and the lip shape change information.

[0009] Optionally, the method for aligning digital population type with voice further includes:

[0010] Constructing the preset speech synthesis model;

[0011] The constructing of the preset speech synthesis model comprises:

[0012] The preset speech synthesis model is trained based on the acquired first speech data including different voice features of different people; wherein the voice features include at least one of speech speed, intonation, tone, language, and accent.

[0013] Optionally, the converting the reply text information into voice information by using a preset speech synthesis model includes:

[0014] The preset speech synthesis model is used to convert the reply text information into speech information that meets the preset sound characteristics.

[0015] Optionally, the method for aligning digital population type with voice further includes:

[0016] Constructing the preset lip shape generation model;

[0017] The constructing of the preset lip shape generation model comprises:

[0018] The preset lip shape generation model is trained based on the acquired video data; wherein the video data is video stream data including lip shape change images of different people and corresponding audio or text.

[0019] Optionally, the method for aligning digital population type with voice further includes:

[0020] Constructing the preset voice and lip alignment model;

[0021] The constructing of the preset voice and lip alignment model comprises:

[0022] According to the acquired video data including the second voice data and the lip shape change picture information, the second voice data of the video data is processed by a preset audio processing model to obtain the parsed sound features of the second voice data; the lip shape change picture information of the video data is processed by a preset lip shape recognition model to obtain the parsed lip shape change information of the lip shape change picture information;

[0023] The preset voice and lip shape alignment model is trained according to the corresponding analyzed sound features and the analyzed lip shape change information.

[0024] Optionally, the input data includes text information and non-text information, and the non-text information includes voice information and / or image information; and determining, based on the input data, reply text information in response to the input data includes:

[0025] When the input data is text information, the text information is input into a preset large language model for inference, and the reply text information is output;

[0026] When the input data is non-text information, the non-text information is converted into the text information, the text information is input into the preset large language model for inference, and the reply text information is outputted.

[0027] Optionally, after constructing the preset voice and lip shape alignment model, the digital mouth shape and voice alignment method further includes:

[0028] Parameter tuning and real-time performance optimization are performed on the preset speech and lip alignment model.

[0029] In order to solve the above problems, the present invention also provides a digital population type and voice alignment device, comprising:

[0030] A reply unit, configured to determine reply text information in response to the input data according to the input data;

[0031] A digital human voice and lip shape generation unit, used to convert the reply text information into voice information by using a preset voice synthesis model, and to convert the reply text information or the voice information into lip shape change information by using a preset lip shape generation model;

[0032] The digital human voice and lip synchronization output unit is used to align the voice information and the lip change information in time series using a preset voice and lip alignment model, and control the digital human to synchronously output the aligned voice information and the lip change information.

[0033] To solve the above problem, the present invention further provides an electronic device, including a memory and a processor;

[0034] The memory is used to store computer programs;

[0035] The processor is used to implement the digital population type and voice alignment method as described above when executing the computer program.

[0036] To solve the above problem, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the method for aligning digital population type and speech as described above is implemented.

[0037] Compared with the prior art, the present invention has the following beneficial effects: the present invention realizes the lip shape drive of the digital human based on machine learning models (or deep learning models) such as a preset speech synthesis model, a preset lip shape generation model and a preset speech and lip shape alignment model, which can improve the efficiency (real-time) of the alignment of the digital human shape and speech while improving the degree of automation of the alignment of the digital human shape and speech, realize timely response to user input, and reduce the cost of the alignment of the digital human shape and speech while ensuring the synchronization of the digital human speech and lip shape. Specifically, the present invention first determines the reply text information in response to the input data according to the corresponding input data to ensure an accurate response to the input; then, based on the reply text information, the corresponding voice information and lip shape change information are generated by a preset voice synthesis model and a preset lip shape generation model to improve the generation efficiency and accuracy of the voice information and lip shape change information, and improve the timeliness (or real-time, efficiency) of the response to the input data; then, the preset voice and lip shape alignment model is used to align the voice information and lip shape change information in a time series and output them synchronously through a digital human to respond to the user's input data visually and auditorily, thereby improving the immersion and interactive experience of the user interacting with the digital human (or the carrier where the digital human is located). BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the flow of the method for aligning digital population type and voice in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of signal transmission of some preset models in an embodiment of the present invention;

[0040] Figure 3 It is a structural block diagram of a device for aligning digital population type and voice in an embodiment of the present invention;

[0041] Figure 4 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0044] Combination Figure 1 , Figure 2As shown, an embodiment of the present invention provides a method for aligning a digital population type with a voice, comprising the following steps:

[0045] Step 100: Determine, based on input data, reply text information in response to the input data.

[0046] Specifically, when corresponding input data is obtained (received), such as input data from a user (which may be text, voice or other forms of input data / information), reply information in response to the input data is determined.

[0047] Exemplarily, in order to improve the accuracy of understanding the content and intent of input data, natural language processing (NLP) technology can be used to process the input data. Therefore, when the input data is text information, the corresponding large language model can be used to process the input data, and the corresponding reply information (recorded as reply text information) with the content of text information is output; when the input data is non-text information (such as voice, image, etc.), the non-text information must first be converted into text information, and then input into the corresponding large language model, and the reply text information is output after being processed by the corresponding large language model. Among them, when the input data is a corresponding question or instruction that can be directly answered through text or voice, the reply text information can be a reply directly corresponding to the input data. For example, when the user asks about the weather of the day, the reply text information can be text information describing the weather of the day; when the input data is a corresponding question or instruction that is difficult to directly answer through text or voice, the reply text information is a corresponding prompt information. For example, when the user needs to perform certain operations on certain devices, the reply text information can be a prompt information such as "x has executed / completed y (where x and y represent the names of the corresponding device and the corresponding operation, respectively)", so as to give users timely feedback and enhance user experience.

[0048] Step 200: Use a preset speech synthesis model to convert the reply text information into speech information, and use a preset lip shape generation model to convert the reply text information or speech information into lip shape change information.

[0049] Specifically, after the reply text information is obtained through step 100, the reply text information can be converted into voice information through a preset (pre-trained) corresponding model for speech synthesis (referred to as a preset speech synthesis model), that is, the preset speech synthesis model generates (or synthesizes) the corresponding voice information for the digital human according to the reply text information. And the reply text information or the converted voice information is converted into lip shape change information through a preset (pre-trained) corresponding model for generating lip shape (i.e., the mouth shape of the digital human) (referred to as a preset lip shape generation model), that is, through the preset lip shape generation model, based on the mapping relationship between speech, text and lip shape within the preset lip shape generation model, the corresponding lip shape change information for the digital human is generated according to the reply text information or the converted voice information. In this way, by converting the reply text information into voice information and lip shape change information, the digital human can have matching lip shape changes while performing sound output, providing the user with a dual sense of reality of vision and hearing, enhancing the user's immersion and interactive experience, etc.

[0050] Step 300: Use a preset voice and lip shape alignment model to align voice information and lip shape change information in time series, and control the digital human to synchronously output the aligned voice information and lip shape change information.

[0051] Specifically, based on the reply text information, a preset (pre-trained) corresponding model (referred to as a preset voice and lip alignment model) is used to align the voice information and lip change information corresponding to the same reply text information in time series, and the voice information and lip change information obtained in step 200 are accurately aligned in time series, so as to achieve synchronization of the voice information and lip change information in time series, thereby ensuring that the digital human can synchronously output the aligned voice information and lip change information, avoiding the sense of disharmony that the digital human's lip shape and voice are not synchronized, so as to significantly improve the user's interaction and usage experience. Exemplarily, by using the trained preset voice and lip alignment model, by adjusting the timestamp of the lip change information sequence output by the preset lip generation model, and / or adjusting the timestamp of the voice information sequence output by the preset speech synthesis model, the corresponding voice information and lip change information are synchronized in time when output by the digital human, so as to achieve the effect of voice and lip alignment. It is worth noting that the digital human's output of voice information is specifically sent out (or played) through a corresponding sound playback device, and the digital human's output of lip shape change information is specifically output (or displayed) through a corresponding display device that displays the digital human.

[0052] In this way, compared with the existing technology that relies on preset rules and manual production by production personnel to achieve digital human population shape and voice alignment, the method of this embodiment is based on machine learning models (or deep learning models) such as preset speech synthesis models, preset lip shape generation models, and preset speech and lip shape alignment models to achieve digital human lip shape drive. It can improve the degree of automation of digital human population shape and voice alignment while improving the efficiency (real-time) of digital human population shape and voice alignment, achieve timely response to user input, and reduce the cost of digital human population shape and voice alignment while ensuring the synchronization of digital human voice and lip shape. Specifically, the method of this embodiment first determines the reply text information in response to the input data according to the corresponding input data to ensure an accurate response to the input; then, based on the reply text information, the corresponding voice information and lip shape change information are generated through a preset voice synthesis model and a preset lip shape generation model to improve the generation efficiency and accuracy of the voice information and lip shape change information, and improve the timeliness (or real-time, efficiency) of the response to the input data; then, the preset voice and lip shape alignment model is used to align the voice information and lip shape change information in a time series and output them synchronously through the digital human to respond to the user's input data visually and auditorily, thereby improving the immersion and interactive experience of the user interacting with the digital human (or the carrier where the digital human is located).

[0053] In addition, the method in this embodiment adopts machine learning models (or deep learning models) such as preset speech synthesis model, preset lip shape generation model and preset speech and lip shape alignment model, which can realize the learning of complex associations such as text, speech and lip shape, and realize the generation of corresponding lip shape changes according to real-time input corresponding data, which can not only ensure that the digital population shape and speech are synchronized in time, but also adjust the shape and change of the lip shape according to the characteristics of pronunciation and intonation in the speech, so as to achieve more accurate lip shape expression, and it has strong flexibility and scalability. Moreover, the model can be trained and optimized according to different speech and lip shape data to meet the needs of virtual digital people with different styles and different character images, which can reduce the production cost of digital population shape and improve production efficiency.

[0054] Optionally, the method further comprises:

[0055] Build a preset speech synthesis model;

[0056] Building a preset speech synthesis model includes:

[0057] A preset speech synthesis model is trained based on the acquired first speech data including different sound features of different people; wherein the sound features include at least one of speech speed, intonation, tone, language, and accent.

[0058] Specifically, in order to improve the applicability of the preset speech synthesis model, that is, to make the preset speech synthesis model better adapt to various types of speech output (i.e., different sound features), and to improve the generalization ability of the preset speech synthesis model, it is necessary to ensure the diversity and richness of the training data set used to train the preset speech synthesis model. Therefore, when constructing the preset speech synthesis model, the corresponding base model (which is a model used as a basis in machine learning or deep learning, which can be a trained model or a simple model structure for subsequent training or expansion; for speech synthesis, the base model can be a simple model including basic speech synthesis functions, and a more complex and accurate speech synthesis model can be obtained by further training or improving it) is trained according to the acquired speech data including different sound features of different people (recorded as the first speech data, which can be acquired in advance as a training data set), so as to train the desired preset speech synthesis model. In some embodiments, the data set used to train the preset speech synthesis model includes not only the first speech data, but also the text data corresponding to the first speech data. The speech data is used to train the model to learn the sound characteristics of the speech, etc., while the text data provides the text content corresponding to the speech, helping the model to learn the correspondence between speech and text, thereby realizing text-to-speech conversion, improving the accuracy and naturalness of the preset speech synthesis model, and ensuring that the preset speech synthesis model can generate a natural and fluent speech output that matches the input text (i.e., the text information input to the preset speech synthesis model) to enhance the expressiveness and interactive effect of the digital human; wherein the text data corresponding to the first speech data can be obtained together with the first speech data, or recognized by a corresponding speech recognition model. The sound characteristics include at least one of speech speed, intonation, tone, language, and accent. Among them, speech speed refers to the speed of speaking, which is generally measured by the number of words or syllables per minute; intonation refers to the rise and fall of the voice when speaking, which is used to express different emotions or functions of speech (such as questions, statements) and other audio characteristics; tone refers to the tone of a specific language (such as the four tones of Chinese), or refers to the emotional attitude or emotional color of the voice when speaking; language refers to the specific language used, such as Chinese, English, Spanish, etc. In some embodiments, in order to further enrich the user's choice (that is, the corresponding language can be set according to the user's needs for the voice output of the digital person), the language classification can be further refined (such as Chinese includes Mandarin, Cantonese, Wu, Xiang, etc.); accent refers to the regional or community characteristics of the pronunciation method, and the accent is usually manifested as a specific pronunciation habit of a specific region or language group (this is considering that the accents of people using the same language are not necessarily exactly the same, and are generally related to the region). In some embodiments, the preset speech synthesis model needs to be based on the reply text information, and through text processing, text analysis, phoneme selection, phoneme conversion, waveform synthesis and other links, the reply text information is converted into speech for realizing the voice output of the virtual digital person.

[0059] In this way, the method of this embodiment combines the different voice features of different people to train the preset speech synthesis model to improve the generalization ability and scope of application of the preset speech synthesis model. For example, it makes the output of the preset speech synthesis model more selective, which can improve the user experience of the corresponding users.

[0060] For ease of understanding, the construction of a deep learning model (such as a preset speech synthesis model) is exemplified below. First, data collection is performed to collect the first voice data; then, considering that there may be environmental and background noise in the first voice data of the mobile phone, the first voice data needs to be preprocessed to remove noise, etc.; then, model selection and design are performed to determine the model architecture (or base model). Commonly used models for deep learning in speech recognition and generation include recurrent neural networks (RNN), long short-term memory networks (LSTM), gated recurrent units (GRU), convolutional neural networks (CNN) and other architectures (models); then, model training is performed, that is, the base model is trained using large-scale (large volume) first voice data. In some embodiments, after model training, model evaluation and optimization are also required to improve the performance and effect of the model.

[0061] Optionally, using a preset speech synthesis model to convert the reply text message into a speech message includes:

[0062] A preset speech synthesis model is used to convert the reply text message into a speech message that meets the preset sound characteristics.

[0063] Specifically, in order to make the voice information output by the digital human more in line with the actual situation or the user's preference and meet the user's personalized needs, the user can be provided with a setting option for the sound feature of the voice information output by the digital human, so that the user can set the sound feature of the voice information output by the digital human, and the set sound feature is recorded as the preset sound feature. When the digital human needs to output voice information, it can output it according to the preset sound feature.

[0064] In this way, by adopting preset sound features, on the one hand, users can customize the sound characteristics of the digital human's voice output according to actual needs and scenarios, so that the voice information output by the digital human is more in line with the actual situation or user preferences; on the other hand, the preset voice synthesis model can generate more natural and fluent voice information according to the preset sound features, thereby improving the quality, efficiency and realism of voice synthesis; on another hand, generating voice information that meets the preset sound features can improve the user's interactive experience with the digital human and enhance the user's immersion and satisfaction.

[0065] Optionally, the method further comprises:

[0066] Construct a preset lip shape generation model;

[0067] Building a preset lip shape generation model includes:

[0068] A preset lip shape generation model is trained based on the acquired video data; wherein the video data is video stream data including lip shape change images of different people and corresponding audio or text.

[0069] Specifically, in order to improve the applicability of the preset lip generation model, that is, to make the lip change information generated by the preset lip generation model accurately correspond to the corresponding text information or voice information, and to improve the generalization ability of the preset lip generation model, it is necessary to ensure the diversity and richness of the training data set used to train the preset lip generation model. Therefore, when constructing the preset lip generation model, the data containing the lip changes are used as the training data set to train the corresponding base model (which can be a simple model containing the basic lip generation function, and a more complex and accurate lip generation model can be obtained by further training or improvement thereof), so as to train the required preset lip generation model. Among them, the data containing the lip changes preferably use video stream data including the lip change pictures of different people and the corresponding audio or the lip change pictures of different people and the corresponding text, so as to reduce the difficulty of obtaining the corresponding training data set, and ensure that the lip change pictures and the corresponding audio or the lip change pictures and the corresponding text can be directly obtained according to the acquired data (the text can be subtitles from the video data or obtained by speech recognition based on the audio of the video data). In some embodiments, the video data may be obtained from public channels such as the Internet and open source data sets, or may be artificially produced (such as recording video data of a person reading aloud according to corresponding text information). In other embodiments, the video data needs to be preprocessed before training the preset lip shape generation model based on the acquired video data to remove noise (such as removing other sounds besides human voice to improve the accuracy of human voice recognition), adjust the frame rate or resolution of the video (in order to accurately obtain the lip shape change information of the person in the video), and verify the validity of the video (such as for videos with pictures, audio and subtitles, voice recognition can be performed based on the audio, and the voice recognition results can be proofread with the subtitles, etc., to ensure that the lip shape change information in the picture can correspond to the audio and subtitles, so as to filter out videos with incorrect correspondence and retain videos with correct correspondence for training data sets (videos with correct correspondence constitute text-lip shape pair data sets, voice-lip shape pair data sets or text-voice-lip shape pair data sets), so as to avoid the incorrectly corresponding videos from affecting the training effect of the model and improve the efficiency and accuracy of model training).

[0070] In this way, the method of this embodiment combines large-scale (large volume) video data including different people's lip shape change pictures and corresponding audio or text to train the preset lip shape generation model, so as to improve the generalization ability and applicability of the preset lip shape generation model.

[0071] For the construction of a preset lip shape generation model, for example, a model design can be performed first to build a variational autoencoder (VAEs) generation model that can receive text data (or voice data) and output corresponding lip shapes, learn the distribution of data from the corresponding input data, and generate new and realistic samples; the received text (reply text information) is segmented and encoded into a vector representation to convert it into a format that the model can understand. The last layer of the model can use a convolutional neural network to generate a lip shape image.

[0072] In the model training phase, a labeled text-lip pair dataset (or speech-lip pair dataset) is used to train the model so that it can learn the mapping relationship from text to lip image (or speech to lip image). First, a dataset containing text (or speech) and corresponding lip images needs to be prepared. This dataset is large and diverse enough to cover as many texts and lip changes as possible, and for each text sample, a corresponding lip image is required as the corresponding label. In addition, a mean square error pixel-level loss function can be used to measure and evaluate the difference between the lip image generated by the model and the real lip image, and optimization algorithms such as gradient descent are used to optimize the model parameters. In this way, during the model training process, the model will continuously adjust its parameters according to the feedback of the loss function to reduce the difference between the generated lip image and the real image, and improve the accuracy of the model's lip generation.

[0073] In this way, after training and optimizing to obtain the final preset lip shape generation model, the preset lip shape generation model can generate a lip shape image corresponding to the input corresponding data based on the input corresponding data and the learned corresponding mapping relationship.

[0074] Optionally, the method further comprises:

[0075] Build preset voice and lip alignment models;

[0076] Building a preset voice and lip alignment model includes:

[0077] A preset voice and lip alignment model is trained based on the acquired video data including the second voice data and the lip shape change picture information.

[0078] The construction of the preset voice and lip shape alignment model aims to achieve the alignment of the corresponding voice information and lip shape change information in the time series, so that the digital human can express the corresponding content more vividly and realistically from the audio-visual aspect, so as to enhance the interaction effect between the user and the digital human and improve the user experience. Specifically, according to the acquired video data including lip shape change picture information of different people (i.e., picture information containing lip shape change pictures of human body, recorded as lip shape change picture information) and corresponding audio information (i.e., voice data or audio data containing human voice information, recorded as second voice data) or text information (which can be obtained based on the second voice data, such as obtained by performing voice recognition on the second voice data), the corresponding base model (which can be a simple model including basic voice and lip shape alignment functions, and a more complex and accurate voice and lip shape alignment model can be obtained by further training or improving it) is trained. During the training process, the voice information (or text information, which can be converted into each other through the corresponding model) and lip shape change information in the video data are learned and trained, so that the model can understand the relationship between voice and lip shape, thereby obtaining a preset voice and lip shape alignment model that can align the corresponding voice information and lip shape change information in a time series.

[0079] Optionally, according to the acquired video data including the second voice data and the lip shape change picture information, training a preset voice and lip shape alignment model includes:

[0080] According to the acquired video data including the second voice data and the lip shape change picture information, the second voice data of the video data is processed by a preset audio processing model to obtain the parsed sound features of the second voice data; the lip shape change picture information of the video data is processed by a preset lip shape recognition model to obtain the parsed lip shape change information of the lip shape change picture information;

[0081] According to the corresponding analyzed sound features and analyzed lip shape change information, a preset speech and lip shape alignment model is trained.

[0082] For the training of the preset voice and lip alignment model, the acquired video data including the second voice data and the lip shape change picture information can be used as the corresponding training set. Specifically, based on the voice data or audio data containing human voice sound information (recorded as second voice data) and the picture information containing the human body's lip shape change picture information (recorded as lip shape change picture information) in the video data, a preset audio processing model (which is a pre-trained model for extracting sound features in audio data, such as a corresponding recurrent neural network, etc.) is used to process the second voice data, and the corresponding sound features corresponding to the second voice data are output (recorded as parsed sound features, which include the corresponding sound features obtained by parsing / processing the second voice data with the preset audio processing model); a preset lip shape recognition model (which is a pre-trained model for extracting features related to lip shape changes, such as a corresponding convolutional neural network) is used to process the lip shape change picture information, and the lip shape change information corresponding to the lip shape change picture information is output (recorded as parsed lip shape change information, or parsed lip shape change feature information, which includes the corresponding lip shape information or lip shape feature information obtained by parsing / processing the lip shape change picture information with the preset lip shape recognition model, such as the shape and texture features of the mouth area, etc.). According to the obtained parsed sound features and parsed lip shape change information, the preset speech and lip shape alignment model can be trained to learn the corresponding relationship between the corresponding speech and lip shape in the time series. In order to improve the efficiency and accuracy of the preset speech and lip shape alignment model training, it is preferred to perform time alignment on the parsed sound features and the parsed lip shape change information to obtain the corresponding parsed sound features and parsed lip shape change information, and then use the corresponding parsed sound features and parsed lip shape change information for the preset speech and lip shape alignment model training to ensure that the model can accurately learn the corresponding relationship between the corresponding speech and lip shape in the time series. In some embodiments, the time alignment annotation of the parsed sound features and the parsed lip shape change information can be performed to obtain the annotation data (used to represent the corresponding relationship between the parsed sound features and the parsed lip shape change information), and the preset voice and lip shape alignment model can be trained based on the parsed sound features, the parsed lip shape change information and the annotation data, wherein the voice and lip shape data are accurately time aligned and annotated by the annotation data, and the annotation includes the start and end time of the voice segment, and the frame number of the corresponding facial lip shape change, and these data are used to train the voice and lip shape alignment model to improve the efficiency and accuracy of the preset voice and lip shape alignment model training. It is worth noting that the data (such as video data, etc.) used for training each corresponding model in this application can be the same data used for training multiple different models to effectively utilize data resources; it can also be different data used for training multiple different models to meet the needs of different models for different scenarios, and by enriching the diversity of data, the dependence of the corresponding model on a specific data set can be reduced, so that the corresponding model has more generalization ability, so that it can better process unknown data and reduce the risk of overfitting.

[0083] Optionally, the preset audio processing model can be trained based on the corresponding audio data in the first voice data and / or video data, and the preset lip shape recognition model can be trained based on the corresponding lip shape change picture information in the video data.

[0084] For ease of understanding, the construction (training) of the preset voice and lip alignment model is exemplified below. Based on video data and deep learning technology, a preset audio processing model such as a recurrent neural network (RNN) is used to process the second voice data in the video data, and the corresponding analytical sound features (of human voice) are extracted from the second voice data; a preset lip recognition model such as a convolutional neural network (CNN) is used to process the lip change picture information in the video data, and the corresponding analytical lip change information is extracted from the lip change picture information. By connecting the input end (or input layer) of the corresponding base model used to train the preset voice and lip alignment model to the output end (or output layer) of the preset audio processing model and the preset lip recognition model, the preset audio processing model and the preset lip recognition model can directly input the analytical sound features and analytical lip change information obtained after processing into the base model, and the base model can be trained according to the analytical sound features and analytical lip change information, and finally the preset voice and lip alignment model is obtained. In some embodiments, the parsed sound features and parsed lip shape change information output by the preset audio processing model and the preset lip shape recognition model are feature fused (such as achieving the correspondence between the parsed sound features and the parsed lip shape change information in time series) and then used for training the corresponding base model. In this way, the fused features are used for training so that the model can learn the accurate correspondence between speech and lip shape. Moreover, a loss function can be defined to measure the difference between the lip shape predicted by the model and the actual lip shape to optimize, evaluate, and guide model training.

[0085] Optionally, for the extraction of lip shape change (feature) information, facial detection and key point positioning technology can be used to extract the positional features of facial images and mouth key points from video data, and a deep learning model (such as a preset lip shape recognition model) can be used to extract the shape and texture features of the mouth area, such as the contour of the lips, the degree of opening and closing, etc.

[0086] Optionally, the input data includes text information and non-text information, and the non-text information includes voice information and / or image information; and determining, according to the input data, reply text information in response to the input data includes:

[0087] When the input data is text information, the text information is input into a preset large language model for inference, and the output is a reply text information;

[0088] When the input data is non-text information, the non-text information is converted into text information, the text information is input into a preset large language model for reasoning, and the reply text information is output.

[0089] Specifically, considering the various ways in which users interact with digital humans (such as text, voice, image, etc.), and in order to improve the accuracy of understanding the content and intention of input data, the method of this embodiment preferably uses natural language processing (NLP) technology to process input data, such as using a pre-trained large language model (referred to as a preset large language model). For the preset large language model, it is necessary to ensure that the data input to the large language model is text information. Therefore, the input data can be divided into text information and non-text information by type. When the input data is text information, the text information can be input into the preset large language model, and the reply text information is output after being processed by the preset large language model, that is, the preset large language model performs semantic understanding and intention analysis on the language of the text information based on the input text information, and generates a reply result (i.e., reply text information) output in the form of text; and when the input data is non-text information, the non-text information must first be converted into text information, and then the converted text information is input into the preset large language model, and the reply text information is output after being processed by the preset large language model. Among them, when the input data is voice information, a pre-trained voice recognition model can be used to convert the voice information into text information for input into a preset large language model; when the input data is image information, a pre-trained image recognition model can be used to convert the image information into text information for input into a preset large language model.

[0090] In this way, the method of this embodiment is applicable to different types of input data (including text information and non-text information), enriching the user's input method and improving the user's interaction and usage experience. In addition, the method of this embodiment uses a preset large language model to process input data, which can effectively improve the accuracy and efficiency of understanding the content and intention of the input data, thereby ensuring that the output content of the digital human can meet the needs of the user and improving the efficiency of effective interaction between the user and the digital human.

[0091] Exemplarily, when the input data is voice information, speech recognition (ASR) technology, such as a speech recognition model, can be used to convert the voice information into a corresponding text sequence. Short-time Fourier transform audio processing technology (STFT) can also be used to extract key spectral features such as the start and end of syllables, pitch changes, volume, and Mel frequency cepstral coefficients (MFCC is used to describe the sound quality of audio) from the text sequence. These features can reflect the sound characteristics of the speech, such as the spectrum and dynamic changes. Among them, the speech recognition model needs to be able to accurately identify the pronunciation and intonation in the voice input data, so as to provide accurate text input for the subsequent large language model.

[0092] Optionally, the method further comprises:

[0093] Build a preset large language model;

[0094] The preset large language models include:

[0095] Based on the acquired language data, a preset large language model is trained.

[0096] Specifically, by obtaining a large amount of language data (text data), which can come from various channels such as the Internet, books, news reports, social media, etc., for training a preset large language model, the diverse training data can improve the generalization ability of the preset large language model. Among them, before the language data is used to train the preset large language model, it is generally necessary to perform pre-processing operations such as cleaning, word segmentation, and removal of stop words on the collected raw data to improve the efficiency of subsequent large language model training. Exemplarily, a preset language large model is constructed through data collection, data preprocessing, model architecture construction, model training, parameter tuning and other technologies; wherein, by training and learning the grammar, semantics and context information of the language, the preset large language model can have a strong semantic understanding and text generation ability, accept the voice text data input by the speech recognition model, etc., and generate natural language text (reply text information) that conforms to the grammar and semantic intentions accordingly, and these text information are transmitted to the speech generation model and the lip generation model for further processing.

[0097] Optionally, after constructing the preset voice and lip alignment model, the method further includes:

[0098] Perform parameter tuning and real-time performance optimization on preset speech and lip alignment models.

[0099] Specifically, the performance of the model is affected by many parameters, including network structure, learning rate, regularization parameters, etc. In order to improve the precision and accuracy of the recognition and generation of the model and achieve more accurate speech and lip-sync expression of the digital human, it is preferred to optimize the parameters of the preset speech and lip-sync model constructed; by adjusting the corresponding parameters of the model, the model can better fit the training data, improve the generalization ability, reduce the risk of overfitting, etc., and further optimize the prediction ability of the model, thereby improving the applicability (i.e., adapting to more different lip shapes and speech changes) and reliability of the preset speech and lip-sync model, making the performance of the digital human more natural and smooth, and improving the user experience and immersion. In some embodiments, regularization, network structure adjustment, learning rate adjustment and other technical means can be used to achieve parameter tuning of the preset speech and lip-sync model.

[0100] For the real-time performance optimization of the preset speech and lip alignment model, such as optimizing the model's operating environment to improve real-time performance, the model's computational efficiency can be optimized by using lightweight network structures, optimization algorithms, parallel computing and other technologies to achieve real-time lip generation and alignment.

[0101] Similarly, after constructing the various preset models mentioned above, parameter tuning and real-time performance optimization are performed on the various preset models to improve the performance of the various preset models.

[0102] Combination Figure 3 As shown, another embodiment of the present invention provides a digital population type and voice alignment device, comprising:

[0103] A reply unit, used to determine reply text information in response to the input data according to the input data;

[0104] A digital human voice and lip shape generation unit, used to convert the reply text information into voice information by using a preset voice synthesis model, and to convert the reply text information or voice information into lip shape change information by using a preset lip shape generation model;

[0105] The digital human speech and lip synchronization output unit is used to align the speech information and lip change information in time series using a preset speech and lip alignment model, and control the digital human to synchronously output the aligned speech information and lip change information.

[0106] In this embodiment, the digital population type and voice alignment device ensures the smoothness and stability of the implementation of the digital population type and voice alignment method through the mutual cooperation between the reply unit, the digital human voice and lip shape generation unit, the digital human voice and lip shape synchronization output unit and other structures. This device realizes the lip shape drive of the digital human based on machine learning models (or deep learning models) such as a preset speech synthesis model, a preset lip shape generation model and a preset voice and lip shape alignment model, which can improve the efficiency (real-time) of the digital population type and voice alignment while improving the automation degree of the digital population type and voice alignment, realize timely response to user input, and reduce the cost of the digital population type and voice alignment while ensuring the synchronization of the digital human voice and lip shape. Specifically, first, based on the corresponding input data, the reply text information responding to the input data is determined to ensure an accurate response to the input; then, based on the reply text information, the corresponding voice information and lip shape change information are generated through a preset voice synthesis model and a preset lip shape generation model to improve the generation efficiency and accuracy of the voice information and lip shape change information, and improve the timeliness (or real-time, efficiency) of the response to the input data; then, the preset voice and lip shape alignment model is used to align the voice information and lip shape change information in a time series and output them synchronously through the digital human, so as to respond to the user's input data visually and auditorily, thereby improving the immersion and interactive experience of the user interacting with the digital human (or the carrier where the digital human is located).

[0107] Combination Figure 4 As shown, another embodiment of the present invention provides an electronic device, including a memory 401 and a processor 402;

[0108] Memory 401, used for storing computer programs;

[0109] The processor 402 is used to implement the above digital population type and voice alignment method when executing the computer program.

[0110] In this embodiment, through the mutual cooperation between the structures such as the memory 401 and the processor 402 of the electronic device, the above-mentioned digital population shape and speech alignment method is smoothly and stably executed, so as to realize the lip shape driving of the digital human based on the machine learning models (or deep learning models) such as the preset speech synthesis model, the preset lip shape generation model and the preset speech and lip shape alignment model. It can improve the efficiency (real-time) of the alignment of the digital population shape and speech while improving the degree of automation of the alignment of the digital population shape and speech, realize timely response to user input, and reduce the cost of aligning the digital population shape and speech while ensuring the synchronization of the digital human speech and lip shape. Specifically, first, based on the corresponding input data, the reply text information responding to the input data is determined to ensure an accurate response to the input; then, based on the reply text information, the corresponding voice information and lip shape change information are generated through a preset voice synthesis model and a preset lip shape generation model to improve the generation efficiency and accuracy of the voice information and lip shape change information, and improve the timeliness (or real-time, efficiency) of the response to the input data; then, the preset voice and lip shape alignment model is used to align the voice information and lip shape change information in a time series and output them synchronously through the digital human, so as to respond to the user's input data visually and auditorily, thereby improving the immersion and interactive experience of the user interacting with the digital human (or the carrier where the digital human is located).

[0111] Another embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor, the above-mentioned method for aligning digital population type and voice is implemented.

[0112] The technical solution of the embodiment of the present invention, in essence or in other words, the part that contributes to the prior art, or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, server, or network device, etc.) or a processor perform all or part of the steps of the method of the embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0113] By storing the computer program corresponding to the method for aligning the digital population type with the voice through a computer-readable storage medium, it is possible to ensure the stability of the computer program corresponding to the method for aligning the digital population type with the voice when it is read and run by the corresponding processor. In this way, the lip-shaped drive of the digital human can be realized based on a machine learning model (or deep learning model) such as a preset speech synthesis model, a preset lip shape generation model, and a preset voice and lip shape alignment model, which can improve the efficiency (real-time) of the alignment of the digital population type with the voice while improving the degree of automation of the alignment of the digital population type with the voice, achieve timely response to user input, and reduce the cost of aligning the digital population type with the voice while ensuring the synchronization of the digital human voice and lip shape. Specifically, first, based on the corresponding input data, the reply text information responding to the input data is determined to ensure an accurate response to the input; then, based on the reply text information, the corresponding voice information and lip shape change information are generated through a preset voice synthesis model and a preset lip shape generation model to improve the generation efficiency and accuracy of the voice information and lip shape change information, and improve the timeliness (or real-time, efficiency) of the response to the input data; then, the preset voice and lip shape alignment model is used to align the voice information and lip shape change information in a time series and output them synchronously through the digital human, so as to respond to the user's input data visually and auditorily, thereby improving the immersion and interactive experience of the user interacting with the digital human (or the carrier where the digital human is located).

[0114] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for aligning digital population type and voice, characterized in that: include: Determining, based on the input data, reply text information in response to the input data; Using a preset speech synthesis model to convert the reply text information into speech information, and using a preset lip shape generation model to convert the reply text information or the speech information into lip shape change information; The voice information and the lip shape change information are aligned in time series using a preset voice and lip shape alignment model, and the digital human is controlled to synchronously output the aligned voice information and the lip shape change information.

2. The method for aligning digital population type and voice according to claim 1, characterized in that: Also includes: Constructing the preset speech synthesis model; The constructing of the preset speech synthesis model comprises: The preset speech synthesis model is trained based on the acquired first speech data including different voice features of different people; wherein the voice features include at least one of speech speed, intonation, tone, language, and accent.

3. The method for aligning digital population type and voice according to claim 1 or 2, characterized in that: The converting the reply text information into voice information by using a preset speech synthesis model comprises: The preset speech synthesis model is used to convert the reply text information into speech information that meets the preset sound characteristics.

4. The method for aligning digital population type and voice according to claim 1 or 2, characterized in that: Also includes: Constructing the preset lip shape generation model; The constructing of the preset lip shape generation model comprises: The preset lip shape generation model is trained based on the acquired video data; wherein the video data is video stream data including lip shape change images of different people and corresponding audio or text.

5. The method for aligning digital population type and voice according to claim 1 or 2, characterized in that: Also includes: Constructing the preset voice and lip alignment model; The constructing of the preset voice and lip alignment model comprises: According to the acquired video data including the second voice data and the lip shape change picture information, the second voice data of the video data is processed by a preset audio processing model to obtain the parsed sound features of the second voice data; the lip shape change picture information of the video data is processed by a preset lip shape recognition model to obtain the parsed lip shape change information of the lip shape change picture information; The preset voice and lip shape alignment model is trained according to the corresponding analyzed sound features and the analyzed lip shape change information.

6. The method for aligning digital population type and voice according to claim 1 or 2, characterized in that: The input data includes text information and non-text information, and the non-text information includes voice information and / or image information; and determining, based on the input data, reply text information in response to the input data includes: When the input data is text information, the text information is input into a preset large language model for inference, and the reply text information is output; When the input data is non-text information, the non-text information is converted into the text information, the text information is input into the preset large language model for inference, and the reply text information is outputted.

7. The method for aligning digital population type and voice according to claim 5, characterized in that: After constructing the preset voice and lip shape alignment model, the digital mouth shape and voice alignment method further includes: Parameter tuning and real-time performance optimization are performed on the preset speech and lip alignment model.

8. A digital population type and voice alignment device, characterized in that: include: A reply unit, configured to determine reply text information in response to the input data according to the input data; A digital human voice and lip shape generation unit, used to convert the reply text information into voice information by using a preset voice synthesis model, and to convert the reply text information or the voice information into lip shape change information by using a preset lip shape generation model; The digital human voice and lip synchronization output unit is used to align the voice information and the lip change information in time series using a preset voice and lip alignment model, and control the digital human to synchronously output the aligned voice information and the lip change information.

9. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the digital population type and voice alignment method as described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the method for aligning digital population type and voice is implemented as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Method for driving digital population type by voice

    CN119864040A

  • Batch video generation method, electronic equipment, storage medium and product

    CN120166267A