Mouth animation sequence generation method and apparatus

Abstract

The present disclosure relates to a mouth shape animation sequence generation method and device. The method comprises: processing input information including input audio and / or input text to obtain a phoneme sequence corresponding to the input information; determining a viseme sequence corresponding to each phoneme in the phoneme sequence based on the phoneme sequence and a matched target mapping table; adjusting the weight of each viseme in the viseme sequence according to the viseme sequence and the corresponding target information to form an adjusted viseme sequence; determining a mouth shape animation corresponding to each viseme in the adjusted viseme sequence based on the target mapping table; and generating a mouth shape animation sequence according to the mouth shape animation. The target mapping table matched with the input information is used to realize the generation of the phoneme sequence based on the input information, the viseme sequence, and the mouth shape animation sequence. The input requirement is low, the mouth shape animation sequence is suitable for different accents and languages, the generation speed is fast, the time is short, the synchronization matching degree with the audio is good, the accuracy is high, and manual debugging is not required, thereby reducing the labor cost.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method and apparatus for generating lip-sync animation sequences. Background Technology

[0002] With the development of industries such as gaming, the interaction between users and virtual characters is becoming increasingly diverse. In scenarios where users can converse with virtual characters, the virtual characters will produce corresponding speech based on the user's voice or text input, and at the same time, the virtual characters will display corresponding lip movements. However, related technologies suffer from various problems, including low synchronization and accuracy between the virtual character's lip movements and the spoken words, limited applicability, high manual costs, and long processing times. Summary of the Invention

[0003] In view of this, the present disclosure proposes a method and apparatus for generating lip-sync animation sequences.

[0004] According to one aspect of this disclosure, a method for generating lip-sync animation sequences is provided, the method comprising:

[0005] The input information is processed to obtain a phoneme sequence corresponding to the input information, wherein the input information includes input audio and / or input text;

[0006] Based on the phoneme sequence and the matched target mapping table, the visual element corresponding to each phoneme in the phoneme sequence is determined to form a visual element sequence. The target mapping table records a first correspondence relationship, which represents the correspondence relationship between phonemes and visual elements.

[0007] Based on the pixel sequence and the corresponding target information, the weight of each pixel in the pixel sequence is adjusted to form an adjusted pixel sequence. The target information includes a target timestamp, which indicates the start and end times of each phoneme in the phoneme sequence.

[0008] Based on the target mapping table, the lip-sync animation corresponding to each of the visual elements in the adjusted visual element sequence is determined. The target mapping table also records a second correspondence, which represents the correspondence between visual elements and lip-sync animation.

[0009] Based on the lip-sync animation, a lip-sync animation sequence corresponding to the input information is generated.

[0010] In one possible implementation, the method further includes:

[0011] Based on the input information, the corresponding speech features are determined;

[0012] Based on the speech features, a matching target mapping table is determined from multiple optional mapping tables;

[0013] The speech features include language, and the first and second correspondences in each of the optional mapping tables match the speech features corresponding to the optional mapping table.

[0014] In one possible implementation, the speech features further include at least one of the following: dialect type, type of overlay object, gender of overlay object, and age of overlay object, wherein the overlay object includes the object that reads the input information.

[0015] In one possible implementation, the input information is processed to obtain a phoneme sequence corresponding to the input information, including:

[0016] When the input information includes input audio and input text, the input text is converted into a corresponding phoneme sequence.

[0017] The input audio is analyzed and processed using a first preset model to obtain a phoneme probability distribution sequence corresponding to the input audio. Based on the phoneme probability distribution sequence and the phoneme sequence, a corresponding target timestamp is generated.

[0018] The phoneme probability distribution sequence is used to indicate the probability vector of the phonemes corresponding to different time steps in the input audio, and the target timestamp is used to indicate the start and end time points of each phoneme in the phoneme sequence in the input audio.

[0019] In one possible implementation, the input information is processed to obtain a phoneme sequence corresponding to the input information, including:

[0020] When the input information includes input audio, the input audio is analyzed and processed using a second preset model to obtain a phoneme probability distribution sequence corresponding to the input audio. Based on the phoneme probability distribution sequence, the corresponding phoneme sequence is determined and a corresponding target timestamp is generated.

[0021] The phoneme probability distribution sequence is used to indicate the probability vector of the phonemes corresponding to different time steps in the input audio, and the target timestamp is used to indicate the start and end time points of each phoneme in the phoneme sequence in the input audio.

[0022] In one possible implementation, the input information is processed to obtain a phoneme sequence corresponding to the input information, including:

[0023] If the input information includes input text, the input text is converted into a corresponding phoneme sequence, and a generated audio corresponding to the input text is generated using a preset audio generation model.

[0024] The generated audio is analyzed and processed using a first preset model to obtain a phoneme probability distribution sequence corresponding to the generated audio. Based on the phoneme probability distribution sequence and the phoneme sequence, a corresponding target timestamp is generated.

[0025] The phoneme probability distribution sequence is used to indicate the probability vector of the phonemes corresponding to different time steps in the generated audio, and the target timestamp is used to indicate the start and end time points of each phoneme in the phoneme sequence in the generated audio.

[0026] In one possible implementation, the target information further includes the volume of each phoneme.

[0027] Based on the pixel sequence and the corresponding target information, the weights of each pixel in the pixel sequence are adjusted to form an adjusted pixel sequence, including:

[0028] Based on the target information, the volume and duration corresponding to each visual element are determined;

[0029] Based on the volume and duration of each visual element, the weight of each visual element in the visual element sequence is adjusted to form an adjusted visual element sequence.

[0030] In one possible implementation, the method further includes:

[0031] The lip-sync animation sequence is superimposed onto a preset target object model to generate a target object animation that includes lip-sync changes; wherein, the target object includes an object that reads the input information.

[0032] In one possible implementation, the method further includes:

[0033] The lip-sync animation in the lip-sync animation sequence is subjected to transition processing;

[0034] The transition process includes at least one of the following:

[0035] The lip-sync animation in the lip-sync animation sequence is adjusted according to the corresponding co-pronunciation rules;

[0036] Insert at least one transition lip-sync animation between adjacent lip-sync animations;

[0037] The amplitude and speed of lip movements in the lip-sync animation sequence are adjusted using a smoothing algorithm.

[0038] According to another aspect of this disclosure, a lip-sync animation sequence generation apparatus is provided, the apparatus comprising:

[0039] An input processing module is used to process input information to obtain a phoneme sequence corresponding to the input information, wherein the input information includes input audio and / or input text;

[0040] The visual element sequence determination module is used to determine the visual element corresponding to each phoneme in the phoneme sequence based on the phoneme sequence and the matched target mapping table, forming a visual element sequence. The target mapping table records a first correspondence relationship, which represents the correspondence relationship between phonemes and visual elements.

[0041] The sequence adjustment module is used to adjust the weight of each visual element in the visual element sequence according to the visual element sequence and the corresponding target information to form an adjusted visual element sequence. The target information includes a target timestamp, which indicates the start time and end time of each phoneme in the phoneme sequence.

[0042] The lip-sync animation determination module is used to determine the lip-sync animation corresponding to each of the visual elements in the adjusted visual element sequence based on the target mapping table. The target mapping table also records a second correspondence, which represents the correspondence between visual elements and lip-sync animations.

[0043] The sequence generation module is used to generate a lip-sync animation sequence corresponding to the input information based on the lip-sync animation.

[0044] According to another aspect of this disclosure, a lip-sync animation sequence generation apparatus is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0045] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0046] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0047] This disclosure provides a method and apparatus for generating lip-sync animation sequences. It generates lip-sync animation sequences using input audio and / or input text as input, reducing the input data requirements for lip-sync animation sequences. A target mapping table matching the input information is used to generate phoneme sequences, then visual sequences, and finally lip-sync animation sequences based on the input information. The generated lip-sync animation sequences are suitable for different accents, languages, and other speech features. Furthermore, the generation speed is fast, the time is short, the synchronization with speech is good, the accuracy is high, and no manual debugging is required, resulting in low labor costs.

[0048] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0049] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0050] Figure 1 A flowchart illustrating a method for generating lip-sync animation sequences according to an embodiment of the present disclosure is shown.

[0051] Figure 2 A flowchart illustrating a method for generating lip-sync animation sequences according to an embodiment of the present disclosure is shown.

[0052] Figure 3 This diagram illustrates a flow chart of one input information processing method in a lip-sync animation sequence generation method according to an embodiment of the present disclosure.

[0053] Figure 4 This is a block diagram illustrating an apparatus 1900 for generating lip-sync animation sequences according to an exemplary embodiment. Detailed Implementation

[0054] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0055] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0056] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0057] In industries such as gaming, users are increasingly demanding higher levels of interaction with virtual characters. Taking games as an example, players' expectations for interactive experiences with characters are growing, with the realism and immersive dialogue experience of NPCs (Non-Player Characters) becoming crucial for enhancing game quality. In the production of NPC lip-sync animation, animators manually create lip-sync sequences frame-by-frame by sensing vowel changes. The production of a complete series can take months or even years, a highly complex process. However, game NPC dialogue tasks require not only precise lip-sync synchronization but also real-time voice dialogue, making manual methods impractical. The continuous development of AI technology has led to its gradual adoption in the gaming field. AI-driven lip-sync technology is mainly divided into two categories: one is the method of generating blend shapes (bs) or meshes, which drives lip-sync through end-to-end bs or mesh generation; the other is a phoneme-based method, which extracts phoneme sequences from speech and combines them with the co-pronunciation rules between phonemes to generate natural lip-sync sequences.

[0058] For example, JALI facial animation technology uses a phoneme-based approach to drive lip movements. By controlling the movements of the jaw and lips, it can create facial animation for characters. This technology is an animator-centric facial animation production system designed to generate high-quality lip-shape results programmatically and embed them into existing animation workflows for further manual fine-tuning by animators. However, for large amounts of dialogue content, manual adjustments increase labor costs and are not conducive to automated workflows. The technology mentions using the Multiple Feature Alignment (MFA) method to align audio and text, outputting the duration of the phoneme sequence. However, actual testing revealed that the alignment results generated by the MFA method are time-consuming and not suitable for real-time lip-sync scenarios. In addition, the MFA alignment method also performed poorly in speech tests with different accents, cross-language, and background noise.

[0059] How to provide a lip-sync animation generation method that is suitable for different accents, languages ​​and other speech features, and that is fast, short in time, highly synchronized with speech and accurate, and low in labor costs is an urgent technical problem to be solved.

[0060] To address the aforementioned technical problems, this disclosure provides a method and apparatus for generating lip-sync animation sequences. By using input audio and / or input text as input, the lip-sync animation sequence is generated, reducing the input data requirements. A target mapping table matching the input information is used to generate the phoneme sequence, then the visual sequence, and finally the lip-sync animation sequence based on the input information. The generated lip-sync animation sequence is suitable for different accents, languages, and other speech features, and the generation speed is fast, the time is short, the synchronization with speech is good, the accuracy is high, and no manual debugging is required, resulting in low labor costs.

[0061] like Figure 1 , Figure 2 As shown, the lip-sync animation generation method provided in this embodiment includes steps S101-S105.

[0062] In step S101, the input information is processed to obtain a phoneme sequence corresponding to the input information, wherein the input information includes input audio and / or input text.

[0063] In this embodiment, the phoneme sequence includes multiple phonemes. A phoneme is the smallest unit of speech defined based on the natural attributes of speech. It is analyzed based on the articulation actions within a syllable, with each action constituting a phoneme. The phoneme sequence includes multiple phonemes, and these phonemes are arranged in the order corresponding to the input information.

[0064] In this embodiment, after receiving input information, the input information can be analyzed to obtain its speech features. These speech features include language, dialect type, type of overlay object, gender and age of the overlay object, and other information representing speech characteristics. The overlay object includes the object reading the input information; this object can be a person, animal, or a non-living cartoon character—any object capable of serving as a mediator for the input information and emitting corresponding speech. This disclosure does not impose any limitations on this. Specifically, the language can be determined directly based on the input text, and the dialect type of some languages ​​can be further determined.

[0065] In some embodiments, voice features can also be determined by combining user input operations, such as determining the type of overlay object, the gender of the overlay object, and the age of the overlay object based on user input.

[0066] In this embodiment, the phonemes differ between different languages. The phonemes may also differ between different dialect types of the same language.

[0067] In this embodiment, the processing of the input information in step S101 differs depending on the type of input information, and will be described separately below.

[0068] Method 1:

[0069] When the input information includes input audio and input text, step S101 may include: converting the input text into a corresponding phoneme sequence; analyzing and processing the input audio using a first preset model to obtain a phoneme probability distribution sequence corresponding to the input audio; and generating a corresponding target timestamp based on the phoneme probability distribution sequence and the phoneme sequence, and using the target timestamp as target information. The phoneme probability distribution sequence is used to indicate the probability vector of the phonemes corresponding to different time steps in the input audio. The target timestamp is used to indicate the start and end time points of each phoneme in the phoneme sequence in the input audio.

[0070] In some embodiments, the first preset model may include a pre-trained Wav2Vec 2.0 model.

[0071] For example, such as Figure 3 As shown, a token generator corresponding to the language and dialect types in speech features can be used to convert input text into tokens and generate corresponding phoneme sequences. A token generator is an algorithm or system that can convert input text into corresponding phoneme sequences and can be implemented using existing technologies.

[0072] like Figure 3 As shown, the input audio can be analyzed and processed based on a pre-trained first Wav2Vec 2.0 model corresponding to the language and dialect type in the speech features and the CTC (Connectionist Temporal Classification) loss function to obtain a phoneme probability distribution sequence for the corresponding input audio. In some embodiments, the process of generating the phoneme probability distribution sequence may include: preprocessing the input audio such as denoising and normalization to obtain audio data; the first Wav2Vec 2.0 model receives the preprocessed audio data and extracts features from it. These features are usually high-dimensional acoustic features that can capture important information of the audio signal. The first Wav2Vec 2.0 model is trained using the CTC loss function, which allows the model to learn to map the input acoustic features to the output phoneme sequence. The first Wav2Vec 2.0 model performs forward propagation on the features of the input audio to generate a phoneme probability distribution for each time step. This distribution represents the probability that the first Wav2Vec 2.0 model believes each phoneme will appear at that time step. The first Wav2Vec 2.0 model outputs a sequence of phoneme probability distributions over consecutive time steps, with each time step corresponding to a probability vector for one phoneme. The phonemes involved in the phoneme probability distribution can be all phonemes applicable to that language and dialect type.

[0073] like Figure 3As shown, after obtaining the phoneme probability distribution sequence and the phoneme sequence, a CTC decoding algorithm (such as the Viterbi algorithm) is used to determine the most likely predicted phoneme sequence corresponding to the input audio based on the phoneme probability distribution sequence. A preliminary timestamp estimation is performed on the predicted phoneme sequence to obtain the prediction of one phoneme for each time step in the input audio. This preliminary timestamp estimation can be implemented using existing technologies such as deep learning. Then, based on the phoneme sequence generated by the marker generator, a connector is used to further optimize the timestamps, determining the start and end times of each phoneme in the input audio, ultimately forming the target timestamp. The connector is an algorithm or model used to align the phoneme sequence with the time axis of the audio signal, thereby determining the start and end times of each phoneme. This process is also known as time alignment or phoneme boundary detection.

[0074] Method 2:

[0075] When the input information includes input audio, step S101 may include: analyzing and processing the input audio using a second preset model to obtain a phoneme probability distribution sequence corresponding to the input audio. Based on the phoneme probability distribution sequence, a corresponding phoneme sequence is determined, and a corresponding target timestamp is generated and used as target information. The target timestamp indicates the start and end times of each phoneme in the phoneme sequence within the input audio. The phoneme probability distribution sequence indicates the probability vector of the phonemes corresponding to different time steps in the input audio.

[0076] In some embodiments, the second preset model may include a pre-trained Wav2Vec 2.0 model.

[0077] For example, the input audio can be preprocessed, such as by denoising and normalization, to obtain audio data. A pre-trained second Wav2Vec 2.0 model (i.e., a second preset model) corresponding to the language and dialect types in the speech features receives the preprocessed audio data and extracts features from it. These features are typically high-dimensional acoustic features that capture important information from the audio signal. The second Wav2Vec 2.0 model is trained using the CTC loss function, which allows the model to learn to map the input acoustic features to the output phoneme sequence. The second Wav2Vec 2.0 model performs forward propagation on the features of the input audio, generating a phoneme probability distribution for each time step, ultimately outputting a phoneme probability distribution sequence, with each time step corresponding to a probability vector for one phoneme. After obtaining the phoneme probability distribution sequence, a CTC decoding algorithm (such as the Viterbi algorithm) is used to determine the most probable phoneme sequence corresponding to the input audio based on the phoneme probability distribution sequence. Timestamp estimation is performed on the phoneme sequence to obtain the prediction of a phoneme corresponding to each time step in the input audio, determine the start and end time points of each phoneme in the input audio, and finally form the target timestamp.

[0078] The difference between Method 2 and Method 1 is that Method 2 uses the phoneme sequence output by the second prediction model as the phoneme sequence used in subsequent steps, while Method 1 uses the phoneme sequence obtained from converting the input text as the phoneme sequence used in subsequent steps. Furthermore, Method 2 directly generates the target timestamp based on the second prediction model; in Method 1, after obtaining the timestamp estimate based on the first prediction model, it optimizes the result using the phoneme sequence obtained from converting the input text to obtain the final target timestamp.

[0079] In some embodiments, the first prediction model and the second prediction model may be the same or different, and this disclosure does not impose any limitations on this. In methods one and two, the inference process using the models (the first preset model and the second preset model) can be implemented using GPU hardware.

[0080] In both Method 1 and Method 2, the trained Wav2Vec 2.0 model is used in conjunction with the CTC loss function to extract the timestamps of phonemes, and GPU hardware is used for network inference, which greatly speeds up the inference efficiency of the alignment model (first preset model and second preset model), enabling the first preset model and the second preset model to meet the real-time requirements, and ensuring high-precision alignment between text and audio, thus ensuring accurate synchronization between the final generated lip-sync animation sequence and the input audio.

[0081] Furthermore, during the training process of the first and second preset models, audio samples with different timbres, pitches, and other pronunciation characteristics, as well as different speech features, can be used for model training. This enables the first and second preset models to accurately extract input audio with different timbres, pitches, and speech features, achieving high model support across different audio formats.

[0082] Method 3:

[0083] When the input information includes input text, step S101 may include: converting the input text into a corresponding phoneme sequence, and generating audio corresponding to the input text using a preset audio generation model. The generated audio is analyzed and processed using a first preset model to obtain a phoneme probability distribution sequence corresponding to the generated audio, and a corresponding target timestamp is generated based on the phoneme probability distribution sequence and the phoneme sequence. The phoneme probability distribution sequence indicates the probability vector of the phonemes corresponding to different time steps in the generated audio, and the target timestamp indicates the start and end times of each phoneme in the phoneme sequence in the generated audio.

[0084] In step S102, based on the phoneme sequence and the matched target mapping table, the visual elements corresponding to each phoneme in the phoneme sequence are determined to form a visual element sequence.

[0085] In this embodiment, a vise is the visual equivalent of a phoneme. It is a phoneme presented visually that depicts the mouth posture, or mouth shape, during pronunciation.

[0086] In this embodiment, the target mapping table records a first correspondence and a second correspondence. The first correspondence represents the correspondence between phonemes and visual pixels. The second correspondence represents the correspondence between visual pixels and lip-sync animation.

[0087] In this embodiment, before step S102, the method may further include: if the corresponding speech features are determined based on the input information, determining a matching target mapping table from multiple optional mapping tables according to the speech features. The speech features include language, and the first and second correspondences in each of the optional mapping tables match the speech features corresponding to that optional mapping table.

[0088] In this embodiment, corresponding optional mapping tables can be constructed in advance based on different speech features, so that the target mapping table can be matched from multiple optional mapping tables based on speech features in the future.

[0089] For example, such as Figure 3The optional mapping table is shown. The corresponding speech features for this optional mapping table can be Chinese, Mandarin, face, female, and 30-40 years old. For Chinese, phonemes can be the initials and finals of pinyin. Based on the actual pronunciation and speech features, a mapping can be created between phonemes, visual pixels, and lip movements (i.e., lip animation). Figure 3 The optional mapping table is shown. Based on... Figure 3 The optional mapping table shown indicates that when pronouncing the phonemes "b", "m", and "p", the visual representation of the mouth shape is that the corners of the mouth are pulled outward, the chin is tightened, and the lips are pursed; when pronouncing the phonemes "o" and "ao", the visual representation of the mouth shape is that the corners of the mouth are contracted inward, the chin is opened downward, and the mouth shape is more rounded.

[0090] In this embodiment, after determining the target mapping table that matches the input information, based on the phoneme sequence and the target mapping table, the visual element corresponding to each phoneme can be found based on the first correspondence between phonemes and visual elements in the target mapping table. Then, a visual element sequence can be formed based on each visual element and the order in which they correspond. The visual element sequence includes multiple visual elements, and the multiple visual elements are arranged in the order corresponding to the input information.

[0091] In step S103, the weights of each visual element in the visual element sequence are adjusted according to the visual element sequence and the corresponding target information to form an adjusted visual element sequence.

[0092] In this embodiment, the target information may further include the volume of each phoneme, and the method may further include: when the input information includes input audio, the first preset model or the second preset model is further used to output the volume corresponding to each phoneme in the input audio. When the input information only includes input text, the volume of each phoneme can be generated based on a volume generation strategy. The volume generation strategy can be set based on the emotion, sentence structure, etc., of the sentences in the input text, and this disclosure does not limit this.

[0093] In this embodiment, step S103 may include: determining the volume and duration corresponding to each visual element based on the target information; adjusting the weight of each visual element in the visual element sequence according to the volume and duration corresponding to each visual element to form an adjusted visual element sequence. In some embodiments, each visual element in the visual element sequence has a preset weight, and then the weight of each visual element is adjusted according to the volume and duration corresponding to each visual element, so that the visual element with high volume and long duration has a higher weight, and the visual element with low volume and short duration has a lower weight.

[0094] In step S104, based on the target mapping table, the lip-sync animation corresponding to each of the visual elements in the adjusted visual element sequence is determined. The target mapping table also records a second correspondence, which represents the correspondence between visual elements and lip-sync animations.

[0095] In step S105, a lip-sync animation sequence corresponding to the input information is generated based on the lip-sync animation.

[0096] In this embodiment, based on the adjusted back pixel sequence, a target mapping table is searched to determine multiple corresponding lip-sync animations, and then a lip-sync animation sequence is generated based on these multiple lip-sync animations. In this way, the generated lip-sync animation sequence can achieve precise synchronization with the audio.

[0097] To make the transitions between lip-sync animations in a lip-sync animation sequence smoother and more natural, one possible implementation method is as follows: Figure 2 As shown, the method may further include: performing transition processing on the lip-sync animation in the lip-sync animation sequence;

[0098] The transition process includes at least one of the following:

[0099] The lip-sync animation in the lip-sync animation sequence is adjusted according to the corresponding co-pronunciation rules;

[0100] Insert at least one transition lip-sync animation between adjacent lip-sync animations;

[0101] The amplitude and speed of lip movements in the lip-sync animation sequence are adjusted using a smoothing algorithm.

[0102] Among them, corresponding co-pronunciation rules can be set in advance according to the characteristics of audio with different speech features, providing corresponding mouth shape basis and coordinated transition for different phonemes, making the transition between mouth shape animations in the mouth shape animation sequence after transition processing smoother and more coherent, and achieving accurate synchronization between mouth shape and speech.

[0103] In this way, by using transition processing, abrupt transitions and unnatural lip movements in the lip-sync animation sequence are eliminated, making the generated lip-sync animation sequence smooth and natural.

[0104] In one possible implementation, the method may further include: superimposing the lip-sync animation sequence onto a preset target object model to generate a target object animation including lip-sync changes; wherein the target object includes an object that reads the input information.

[0105] In this implementation, a lip-sync animation sequence is overlaid onto a preset target object model. Rendering then creates an animation of the target object, in which the lip movements of the target object change according to the lip-sync animation sequence. This allows for synchronized playback of the corresponding audio while the target object animation is playing, achieving precise correspondence and synchronized playback between lip movements and audio. Furthermore, it enables precise and synchronized playback of audio and lip movements with different speech characteristics, making it widely applicable and suitable for diverse environments.

[0106] This disclosure also provides a lip-sync animation sequence generation apparatus, the apparatus comprising:

[0107] An input processing module is used to process input information to obtain a phoneme sequence corresponding to the input information, wherein the input information includes input audio and / or input text;

[0108] The visual element sequence determination module is used to determine the visual element corresponding to each phoneme in the phoneme sequence based on the phoneme sequence and the matched target mapping table, forming a visual element sequence. The target mapping table records a first correspondence relationship, which represents the correspondence relationship between phonemes and visual elements.

[0109] The sequence adjustment module is used to adjust the weight of each visual element in the visual element sequence according to the visual element sequence and the corresponding target information to form an adjusted visual element sequence. The target information includes a target timestamp, which indicates the start time and end time of each phoneme in the phoneme sequence.

[0110] The lip-sync animation determination module is used to determine the lip-sync animation corresponding to each of the visual elements in the adjusted visual element sequence based on the target mapping table. The target mapping table also records a second correspondence, which represents the correspondence between visual elements and lip-sync animations.

[0111] The sequence generation module is used to generate a lip-sync animation sequence corresponding to the input information based on the lip-sync animation.

[0112] In one possible implementation, the device may further include:

[0113] The mapping table selection module is used to determine the corresponding speech features based on the input information; and to determine the matching target mapping table from multiple optional mapping tables according to the speech features.

[0114] The speech features include language, and the first and second correspondences in each of the optional mapping tables match the speech features corresponding to the optional mapping table.

[0115] In one possible implementation, the speech features further include at least one of the following: dialect type, type of overlay object, gender of overlay object, and age of overlay object, wherein the overlay object includes the object that reads the input information.

[0116] In one possible implementation, the input processing module includes:

[0117] The first input processing submodule is configured to convert the input text into a corresponding phoneme sequence when the input information includes input audio and input text.

[0118] The input audio is analyzed and processed using a first preset model to obtain a phoneme probability distribution sequence corresponding to the input audio. Based on the phoneme probability distribution sequence and the phoneme sequence, a corresponding target timestamp is generated.

[0119] The phoneme probability distribution sequence is used to indicate the probability vector of the phonemes corresponding to different time steps in the input audio, and the target timestamp is used to indicate the start and end time points of each phoneme in the phoneme sequence in the input audio.

[0120] In one possible implementation, the input processing module includes:

[0121] The second input processing submodule is used to analyze and process the input audio using a second preset model when the input information includes input audio, to obtain a phoneme probability distribution sequence corresponding to the input audio, and to determine the corresponding phoneme sequence and generate the corresponding target timestamp based on the phoneme probability distribution sequence.

[0122] The phoneme probability distribution sequence is used to indicate the probability vector of the phonemes corresponding to different time steps in the input audio, and the target timestamp is used to indicate the start and end time points of each phoneme in the phoneme sequence in the input audio.

[0123] In one possible implementation, the input processing module includes:

[0124] The third input processing submodule is used to convert the input text into a corresponding phoneme sequence when the input information includes input text, and to generate audio corresponding to the input text using a preset audio generation model.

[0125] The generated audio is analyzed and processed using a first preset model to obtain a phoneme probability distribution sequence corresponding to the generated audio. Based on the phoneme probability distribution sequence and the phoneme sequence, a corresponding target timestamp is generated.

[0126] The phoneme probability distribution sequence is used to indicate the probability vector of the phonemes corresponding to different time steps in the generated audio, and the target timestamp is used to indicate the start and end time points of each phoneme in the phoneme sequence in the generated audio.

[0127] In one possible implementation, the target information further includes the volume of each phoneme, and the sequence adjustment module includes:

[0128] The volume and duration determination submodule is used to determine the volume and duration corresponding to each visual element based on the target information.

[0129] The adjustment submodule is used to adjust the weight of each visual element in the visual element sequence according to the volume and duration of each visual element, so as to form an adjusted visual element sequence.

[0130] In one possible implementation, the device further includes:

[0131] An animation generation module is used to overlay the lip-shape animation sequence onto a preset target object model to generate a target object animation that includes lip-shape changes.

[0132] The target object includes the object that reads the input information aloud.

[0133] In one possible implementation, the device further includes:

[0134] A transition processing module is used to perform transition processing on the lip-sync animation in the lip-sync animation sequence;

[0135] The transition process includes at least one of the following:

[0136] The lip-sync animation in the lip-sync animation sequence is adjusted according to the corresponding co-pronunciation rules;

[0137] Insert at least one transition lip-sync animation between adjacent lip-sync animations;

[0138] The amplitude and speed of lip movements in the lip-sync animation sequence are adjusted using a smoothing algorithm.

[0139] It should be noted that although the above embodiments have been used as examples to illustrate the method and apparatus for generating lip-sync animation sequences, those skilled in the art will understand that this disclosure is not limited thereto. In fact, users can flexibly set each step and module according to their personal preferences and / or actual application scenarios, as long as it conforms to the technical solution of this disclosure.

[0140] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0141] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0142] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0143] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0144] Figure 4 This is a block diagram illustrating an apparatus 1900 for generating lip-sync animation sequences according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 4 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0145] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0146] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0147] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0148] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0149] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0150] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0151] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0152] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0153] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0155] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of generating a mouth animation sequence, characterized by, The method comprises: processing input information to obtain a phoneme sequence corresponding to the input information, the input information comprising input audio and / or input text; determining, based on the phoneme sequence and a matched target mapping table, a viseme corresponding to each phoneme in the phoneme sequence to form a viseme sequence, the target mapping table recording a first correspondence relationship representing a correspondence relationship between phonemes and visemes; adjusting the weight of each viseme in the viseme sequence according to the viseme sequence and corresponding target information to form an adjusted viseme sequence, the target information comprising a target timestamp indicating a starting time point and an ending time point of each phoneme in the phoneme sequence; determining, based on the target mapping table, a mouth shape animation corresponding to each viseme in the adjusted viseme sequence, the target mapping table further recording a second correspondence relationship representing a correspondence relationship between visemes and mouth shape animations; generating a mouth shape animation sequence corresponding to the input information according to the mouth shape animation; wherein the method further comprises determining a corresponding speech feature based on the input information; determining a matched target mapping table from a plurality of selectable mapping tables according to the speech feature; the speech feature comprises a language, and the first correspondence relationship and the second correspondence relationship in each of the selectable mapping tables match the speech feature corresponding to the selectable mapping table; wherein the speech feature further comprises at least one of the following: a dialect type, a type of superimposed object, a gender of a superimposed object, and an age, the superimposed object comprising an object reading the input information; wherein the target information further comprises a volume of each phoneme, and adjusting the weight of each viseme in the viseme sequence according to the viseme sequence and corresponding target information to form an adjusted viseme sequence comprises: determining, based on the target information, a volume and a duration of each viseme; adjusting the weight of each viseme in the viseme sequence according to the volume and the duration of each viseme to form an adjusted viseme sequence.

2. The method of claim 1, wherein, processing input information to obtain a phoneme sequence corresponding to the input information comprises: in the case where the input information comprises input audio and input text, converting the input text into a corresponding phoneme sequence, analyzing and processing the input audio using a first preset model to obtain a phoneme probability distribution sequence corresponding to the input audio, and generating a corresponding target timestamp based on the phoneme probability distribution sequence and the phoneme sequence, wherein the phoneme probability distribution sequence is used to indicate a probability vector of a phoneme corresponding to different time steps in the input audio, and the target timestamp is used to indicate a starting time point and an ending time point of each phoneme in the phoneme sequence in the input audio.

3. The method of claim 1, wherein, processing input information to obtain a phoneme sequence corresponding to the input information comprises: In a case where the input information comprises input audio, the input audio is analyzed and processed by using a second preset model to obtain a phoneme probability distribution sequence corresponding to the input audio, and a corresponding phoneme sequence and a corresponding target timestamp are determined according to the phoneme probability distribution sequence, wherein the phoneme probability distribution sequence is used to indicate a probability vector of a phoneme corresponding to different time steps in the input audio, and the target timestamp is used to indicate a starting time point and an ending time point of each phoneme in the phoneme sequence in the input audio.

4. The method of claim 1, wherein, The input information is processed to obtain a phoneme sequence corresponding to the input information, comprising: In a case where the input information comprises input text, the input text is converted into a corresponding phoneme sequence, and a generated audio corresponding to the input text is generated by using a preset audio generation model, the generated audio is analyzed and processed by using a first preset model to obtain a phoneme probability distribution sequence corresponding to the generated audio, and a corresponding target timestamp is generated based on the phoneme probability distribution sequence and the phoneme sequence, wherein the phoneme probability distribution sequence is used to indicate a probability vector of a phoneme corresponding to different time steps in the generated audio, and the target timestamp is used to indicate a starting time point and an ending time point of each phoneme in the phoneme sequence in the generated audio.

5. The method of claim 1, wherein, The method further comprises: superimposing the lip movement animation sequence into a preset target object model to generate a target object animation comprising lip movement changes; wherein the target object comprises an object reading the input information.

6. The method of claim 1, wherein, The method further comprises: transition processing is performed on the lip movement animations in the lip movement animation sequence; wherein the transition processing comprises at least one of the following: adjusting the lip movement animations in the lip movement animation sequence according to a corresponding coarticulation rule; inserting at least one transition lip movement animation between adjacent lip movement animations; adjusting the change amplitude and speed of the lip movement in the lip movement animation sequence by using a smoothing algorithm.

7. An apparatus for generating a mouth animation sequence, characterized by The device comprises: an input processing module configured to process input information to obtain a phoneme sequence corresponding to the input information, the input information comprising input audio and / or input text; a phoneme sequence determination module configured to determine corresponding phonemes in the phoneme sequence based on the phoneme sequence and a matched target mapping table to form a phoneme sequence, the target mapping table recording a first correspondence relationship, and the first correspondence relationship representing a correspondence relationship between phonemes and phonemes; a sequence adjustment module configured to adjust weights of each phoneme in the phoneme sequence according to the phoneme sequence and corresponding target information to form an adjusted phoneme sequence, the target information comprising a target timestamp, and the target timestamp indicating a starting time point and an ending time point of each phoneme in the phoneme sequence; a lip movement animation determination module configured to determine a lip movement animation corresponding to each phoneme in the adjusted phoneme sequence based on the target mapping table, and the target mapping table further recording a second correspondence relationship, and the second correspondence relationship representing a correspondence relationship between phonemes and lip movement animations. The sequence generation module is configured to generate a mouth shape animation sequence corresponding to the input information according to the mouth shape animation. The device further comprises a mapping table selection module configured to determine a corresponding speech feature based on the input information, and determine a target mapping table matched from a plurality of selectable mapping tables according to the speech feature. The speech feature comprises a language type. The first corresponding relationship and the second corresponding relationship in each of the selectable mapping tables are matched with the speech feature corresponding to the selectable mapping table. The speech feature further comprises at least one of the following: a dialect type, a type of superimposed object, a gender of the superimposed object, and an age. The superimposed object comprises an object reading the input information. The target information further comprises a volume of each of the phonemes. The sequence adjustment module comprises: A volume and duration determination sub-module configured to determine a volume and a duration corresponding to each of the phonemes based on the target information. An adjustment sub-module configured to adjust a weight of each of the phonemes in the phoneme sequence according to the volume and the duration corresponding to each of the phonemes, to form an adjusted phoneme sequence.

8. An apparatus for generating a mouth animation sequence, characterized by The device comprises: A processor; A memory for storing processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 6 when executing the instructions stored in the memory.

9. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 6.

Images