Conversation mouth shape generation method and system, electronic equipment and storage medium

Abstract

The invention discloses a dialogue mouth shape generation method and system, electronic equipment and a storage medium. The dialogue mouth shape generation method comprises the steps of obtaining an original dialogue text, an original dialogue sound source and a phoneme set corresponding to the type of the original dialogue text; performing phoneme conversion on the original dialogue text to obtain a dialogue phoneme text, and performing volume analysis on an original dialogue sound source to generate a volume labeling file; performing phoneme fitting on the original dialogue sound source, the dialogue phoneme text and the phoneme set to generate a phoneme annotation file; obtaining a role model and a plurality of target shapes corresponding to the role model; setting an initial weight value corresponding to the target shape according to the phoneme set; performing weight correction on the initial weight value of the target shape according to the volume labeling file to obtain animation weight data; and generating a mouth shape corresponding to the role model according to the animation weight data. According to the embodiment of the invention, phoneme labeling and volume labeling information can be comprehensively considered, and the matching degree of the animation mouth shape and pronunciation is improved, so that the mouth shape animation generation effect is enhanced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, system, electronic device and storage medium for generating dialogue lip shapes. Background Art

[0002] In 3D animation, virtual reality (VR), augmented reality (AR) and game development, the natural dialogue performance of virtual characters is crucial to improving user experience. Traditional lip animation production methods often rely on manual keyframe adjustment, which is not only time-consuming and labor-intensive, but also difficult to achieve highly realistic effects.

[0003] In recent years, there have been technologies for automatically generating lip animations based on phoneme recognition or phoneme alignment. However, the lip animation generation methods using phoneme recognition or phoneme alignment are too dependent on the accuracy of phoneme recognition for the effect of animation generation. The current easy-to-implement phoneme recognition technology is still difficult to achieve absolute accuracy. When the phoneme recognition effect is not good, the phoneme alignment results will have obvious deviations and significantly affect the lip animation generation effect. Secondly, the current methods ignore the impact of volume changes on the naturalness of the lip shape, resulting in a mismatch between the lip shape and the emotional expression of the voice in some cases. Summary of the invention

[0004] The embodiments of the present application provide a method, system, electronic device and storage medium for generating dialogue lip shapes, which can comprehensively consider phoneme annotation and volume annotation information, improve the matching degree between animated lip shapes and pronunciation, and thus enhance the lip shape animation generation effect.

[0005] In a first aspect, an embodiment of the present application provides a method for generating dialogue lip shapes, the method comprising:

[0006] Acquire an original dialogue text, an original dialogue sound source, and a phoneme set corresponding to the type of the original dialogue text;

[0007] Performing phoneme conversion on the original dialogue text to obtain a dialogue phoneme text, and performing volume analysis on the original dialogue sound source to generate a volume annotation file;

[0008] Performing phoneme fitting on the original dialogue sound source, the dialogue phoneme text and the phoneme set to generate a phoneme annotation file;

[0009] Obtaining a character model in a preset fusion deformation set, and obtaining a plurality of target shapes corresponding to the character model;

[0010] For each of the target shapes, setting an initial weight value corresponding to the target shape according to the phoneme set;

[0011] Performing weight correction on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data;

[0012] A lip shape corresponding to the character model is generated according to the animation weight data.

[0013] In some embodiments, the converting the original dialogue text into phonemes to obtain the dialogue phoneme text includes:

[0014] Determining a text category of the original conversation text, and determining a text processing dictionary according to the text category;

[0015] Performing word recognition on the original dialogue text through a preset associated word dictionary to obtain monophonetic words and polyphonetic words;

[0016] Performing pinyin search on the monophonetic word and the polyphonetic word by using a preset search algorithm to determine a first phoneme text corresponding to the monophonetic word and a second phoneme text corresponding to the polyphonetic word in the text processing dictionary;

[0017] A dialogue phoneme text is generated according to the first phoneme text and the second phoneme text.

[0018] In some embodiments, the performing volume analysis on the original dialogue sound source to generate a volume annotation file includes:

[0019] Determining the overall sampling duration of the original dialogue audio source, and setting the sampling step length according to the overall sampling duration;

[0020] Performing audio sampling on the original dialogue sound source according to the sampling step length to obtain a plurality of sampling frames and volume values ​​corresponding to the sampling frames;

[0021] The sampling frames and the volume values ​​corresponding to the sampling frames are used as key frames, and all the key frames are normalized to generate a volume annotation file.

[0022] In some embodiments, performing phoneme fitting on the original dialogue sound source, the dialogue phoneme text, and the phoneme set to generate a phoneme annotation file includes:

[0023] Performing text annotation on the dialogue phoneme text to obtain corpus data;

[0024] Extracting features from the original conversation sound source using a preset speech recognition tool to obtain a feature vector;

[0025] Inputting the feature vector and the corpus data into the speech recognition tool for model training to obtain a language acoustic model, and constructing a pronunciation dictionary corresponding to the language acoustic model according to the phoneme set;

[0026] Training the speech recognition model in the speech recognition tool according to the language acoustic model and the pronunciation dictionary to obtain a pre-trained speech recognition model;

[0027] For each dialogue phoneme in the dialogue phoneme text, obtaining a start timestamp and an end timestamp of the dialogue phoneme, and performing a maximum likelihood path fitting solution on the dialogue phoneme through the pre-trained speech recognition model to obtain a phoneme fitting degree;

[0028] A phoneme annotation file is generated according to the start timestamp, the end timestamp and the phoneme fit.

[0029] In some embodiments, performing weight correction on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data includes:

[0030] For each target phoneme in the phoneme annotation file, determining a start normalized volume value and an end normalized volume value of the target phoneme according to the volume annotation file;

[0031] Determining a volume mean value according to the start normalized volume value and the end normalized volume value;

[0032] Performing a square root process on the volume mean to obtain a correction multiplier corresponding to the target phoneme;

[0033] Animation weight data is determined according to the correction multiplier and the initial weight value.

[0034] In some embodiments, after weight correction is performed on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data, the method further includes:

[0035] Merging the data in the volume annotation file and the data in the phoneme annotation file into the same time axis to obtain a target time axis;

[0036] Performing interval screening on the target time axis according to a preset normalized volume threshold to screen out a target segment greater than the normalized volume threshold;

[0037] When the target segment does not cover the phoneme, determining a first segment and a second segment adjacent to the target segment, and calculating a segment normalized volume mean of the target segment;

[0038] Obtaining a first degree of fit corresponding to the first segment and a second degree of fit corresponding to the second segment;

[0039] Determining a segment target value according to the first fit degree, the second fit degree, and the segment normalized volume mean;

[0040] When the segment target value is less than a preset constant value, the phonemes on the target time axis are aligned and corrected.

[0041] In some embodiments, the aligning and correcting the phonemes on the target time axis includes:

[0042] Determine the phoneme original alignment time point according to the phoneme annotation file, wherein the phoneme original alignment time point is the midpoint between the timestamp of the beginning of the phoneme and the timestamp of the ending of the phoneme;

[0043] Determining a segment center time point of the target segment;

[0044] Determine the time median of the phoneme original alignment time point and the segment center time point, and use the time median as the offset midpoint;

[0045] The position of the phoneme on the target time axis is corrected according to the offset midpoint.

[0046] In a second aspect, the present application also provides a dialogue lip shape generation system, the system comprising:

[0047] A data acquisition module, used to acquire the original dialogue text, the original dialogue sound source and a phoneme set corresponding to the type of the original dialogue text;

[0048] A data analysis module, used to perform phoneme conversion on the original dialogue text to obtain a dialogue phoneme text, and perform volume analysis on the original dialogue sound source to generate a volume annotation file;

[0049] A phoneme fitting module, used for performing phoneme fitting on the original dialogue sound source, the dialogue phoneme text and the phoneme set to generate a phoneme annotation file;

[0050] A model acquisition module, used to acquire a character model in a preset fusion deformation set, and acquire multiple target shapes corresponding to the character model;

[0051] A weight setting module, configured to set, for each of the target shapes, an initial weight value corresponding to the target shape according to the phoneme set;

[0052] A weight correction module, used for performing weight correction on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data;

[0053] A lip shape generation module is used to generate a lip shape corresponding to the character model according to the animation weight data.

[0054] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for generating dialogue lip shapes as described in the first aspect is implemented.

[0055] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for generating dialogue lip shapes as described in the first aspect.

[0056] The method for generating dialogue lip shapes provided in the embodiment of the present application has at least the following beneficial effects: first, an original dialogue text, an original dialogue sound source, and a phoneme set corresponding to the type of the original dialogue text are obtained, wherein the phoneme set stores syllable structures and phonological features corresponding to the type of the original dialogue text; then, the original dialogue text is converted into phonemes, so that more subtle pronunciation differences can be captured to obtain a dialogue phoneme text, thereby reducing recognition errors caused by different accents and pronunciation habits; and a volume analysis is performed on the original dialogue sound source to analyze the volume changes in the original dialogue sound source, and the volume level is recorded according to the time axis to generate a volume annotation file; by analyzing the volume changes, different parts in the dialogue sound source can be more accurately identified; then, phoneme fitting is performed on the original dialogue sound source, the dialogue phoneme text, and the phoneme set, and the fitting degree of each phoneme is analyzed to generate a phoneme annotation file, thereby providing more detailed voice features; and by analyzing the phoneme fitting degree, voice data can be more accurately annotated, thereby improving the efficiency and quality of data annotation. Afterwards, the character model in the preset fusion deformation set is obtained, and multiple target shapes corresponding to the character model are obtained, so that different weight values ​​are assigned to different target shapes in the future, and the mouth shape corresponding to the target shape is further generated to achieve the subdivision of the character mouth shape. For each target shape, the initial weight value corresponding to the target shape is set according to the phoneme set, so that the character mouth shape effect of different language dialogues can be accurately generated, and then the initial weight value of the target shape is weighted according to the volume annotation file. The alignment result with low fitting degree can be corrected by the volume annotation file, so as to significantly improve the situation where the pronunciation area lacks the corresponding mouth shape when the phoneme recognition effect is not good, and the animation weight data is obtained. The animation weight is calculated by mixing the volume annotation data, which can better present the subtle differences in the speaker's mouth shape at different volumes, and then the mouth shape corresponding to the character model is generated according to the animation weight data, so that the transition of the character dialogue mouth shape appears more natural and smooth. The embodiment of the present application generates animation weight data by comprehensively considering the phoneme annotation and volume annotation information, so as to accurately simulate the changes in human mouth shape at different volumes and enhance the realism and expressiveness of the character dialogue.

[0057] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the examples of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0059] Figure 1 is a flowchart of a specific method of the method for generating dialogue lip shapes provided in an embodiment of the present application;

[0060] Figure 2 This is a specific flow chart of performing phoneme conversion on the original dialogue text provided by an embodiment of the present application;

[0061] Figure 3 is a specific flow chart of performing volume analysis on an original dialogue sound source provided by an embodiment of the present application;

[0062] Figure 4 It is a specific flow chart of performing phoneme fitting on the original dialogue sound source, dialogue phoneme text and phoneme set provided by the embodiment of the present application;

[0063] Figure 5 It is a specific flow chart of weight correction of the initial weight value of the target shape according to the volume annotation file provided in an embodiment of the present application;

[0064] Figure 6 is a specific flow chart of a method for generating dialogue lip shapes provided in another embodiment of the present application;

[0065] Figure 7 It is a specific flow chart of aligning and correcting phonemes on a target time axis provided by an embodiment of the present application;

[0066] Figure 8 is a structural schematic diagram of a dialogue lip shape generation system provided in an embodiment of the present application;

[0067] Fig. 9 It is a schematic diagram of the overall framework provided by an example of this application;

[0068] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] The method for generating lip shapes for a conversation provided in the embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a smart watch, etc.; the server can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms; the software can be an application that implements the above method, etc., but is not limited to the above forms.

[0073] Embodiments of the present application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] In 3D animation, virtual reality (VR), augmented reality (AR) and game development, the natural dialogue performance of virtual characters is crucial to improving user experience. Traditional lip animation production methods often rely on manual keyframe adjustment, which is not only time-consuming and labor-intensive, but also difficult to achieve highly realistic effects.

[0075] In recent years, there have been technologies for automatically generating lip animations based on phoneme recognition or phoneme alignment. However, the lip animation generation methods using phoneme recognition or phoneme alignment are too dependent on the accuracy of phoneme recognition for the effect of animation generation. The current easy-to-implement phoneme recognition technology is still difficult to achieve absolute accuracy. When the phoneme recognition effect is not good, the phoneme alignment results will have obvious deviations and significantly affect the lip animation generation effect. Secondly, the current methods ignore the impact of volume changes on the naturalness of the lip shape, resulting in a mismatch between the lip shape and the emotional expression of the voice in some cases.

[0076] In order to solve the above problems, the embodiments of the present application provide a method, system, electronic device and storage medium for generating dialogue lip shapes. First, an original dialogue text, an original dialogue sound source and a phoneme set corresponding to the type of the original dialogue text are obtained, wherein the phoneme set stores the syllable structure and phonological features corresponding to the type of the original dialogue text. Then, the original dialogue text is converted into phonemes, which can capture more subtle pronunciation differences and obtain a dialogue phoneme text, thereby reducing recognition errors caused by different accents and pronunciation habits. The original dialogue sound source is analyzed for volume, and the volume changes in the original dialogue sound source are analyzed. The volume level is recorded according to the time axis to generate a volume annotation file. By analyzing the volume changes, different parts in the dialogue sound source can be more accurately identified. Then, the original dialogue sound source, the dialogue phoneme text and the phoneme set are fitted with phones, the fitting degree of each phoneme is analyzed, and a phoneme annotation file is generated, thereby providing more detailed voice features. By analyzing the phoneme fitting degree, voice data can be annotated more accurately, thereby improving the efficiency and quality of data annotation. Afterwards, the character model in the preset fusion deformation set is obtained, and multiple target shapes corresponding to the character model are obtained, so that different weight values ​​are assigned to different target shapes in the future, and the mouth shape corresponding to the target shape is further generated to achieve the subdivision of the character mouth shape. For each target shape, the initial weight value corresponding to the target shape is set according to the phoneme set, so that the character mouth shape effect of different language dialogues can be accurately generated, and then the initial weight value of the target shape is weighted according to the volume annotation file. The alignment result with low fitting degree can be corrected by the volume annotation file, so as to significantly improve the situation where the pronunciation area lacks the corresponding mouth shape when the phoneme recognition effect is not good, and the animation weight data is obtained. The animation weight is calculated by mixing the volume annotation data, which can better present the subtle differences in the speaker's mouth shape at different volumes, and then the mouth shape corresponding to the character model is generated according to the animation weight data, so that the transition of the character dialogue mouth shape appears more natural and smooth. The embodiment of the present application generates animation weight data by comprehensively considering the phoneme annotation and volume annotation information, so as to accurately simulate the changes in human mouth shape at different volumes and enhance the realism and expressiveness of the character dialogue.

[0077] Reference Figure 1 , Figure 1 1 is a flowchart of a specific method of the method for generating lip shape for dialogue provided by an embodiment of the present application. In some embodiments, the method for generating lip shape for dialogue includes but is not limited to steps S101 to S107.

[0078] Step S101, obtaining the original dialogue text, the original dialogue sound source, and the phoneme set corresponding to the type of the original dialogue text.

[0079] In step S101 of some embodiments, since different types of languages ​​correspond to different phonemes, and different languages ​​have great differences in phoneme recognition and lip shape correspondence, the embodiments of the present application first obtain the original dialogue text, the original dialogue sound source, and the phoneme set corresponding to the type of the original dialogue text, and the phoneme set stores the syllable structure and phonological features corresponding to the type of the original dialogue text. For example, for Chinese speech language, the phoneme set stores all pinyin initials and all pinyin finals; for English speech language, the phoneme set stores all consonants and vowels, etc., which is convenient for the subsequent generation of corresponding lip shapes for different voices and improves the accuracy of lip shape generation.

[0080] Step S102, performing phoneme conversion on the original dialogue text to obtain a dialogue phoneme text, and performing volume analysis on the original dialogue sound source to generate a volume annotation file.

[0081] In step S102 of some embodiments, the original dialogue text is converted into phonemes, so that the converted dialogue phoneme text corresponds to the original dialogue text, and a smooth and natural voice output is generated, which can handle multiple languages ​​and dialects and provide high-precision phoneme conversion. The volume of the original dialogue sound source is analyzed and a volume annotation file is generated, thereby enhancing the consistency of the audio and further improving the playback quality and consistency of the original dialogue sound source.

[0082] Step S103, performing phoneme fitting on the original dialogue sound source, the dialogue phoneme text and the phoneme set to generate a phoneme annotation file.

[0083] In step S103 of some embodiments, phoneme fitting is performed on the original dialogue sound source, dialogue phoneme text, and phoneme set to obtain the degree of fitting corresponding to each dialogue phoneme, so that the generated phoneme annotation file includes the timestamps corresponding to the start and end of each dialogue phoneme, and also includes the degree of fitting corresponding to each dialogue phoneme, so as to handle the inherent dependencies of time series data.

[0084] Step S104, obtaining a character model in a preset fusion deformation set, and obtaining a plurality of target shapes corresponding to the character model.

[0085] In step S104 of some embodiments, a character model in a preset fusion deformation set is obtained, and multiple target shapes corresponding to the character model are obtained, wherein the target shapes are related to the facial deformation of the character model. By changing the target shapes, the lip shape of the character model can be changed, thereby facilitating the subsequent generation of lip shapes corresponding to the original dialogue text and the original dialogue sound source.

[0086] It should be noted that the fused deformation set in the embodiment of the present application is a BlendShape set. BlendShape is a technology for generating 3D animation through mixed deformation. BlendShape can read the vertex index and position information in the target shape, and interpolate each vertex of the basic mesh to the target shape according to the weight of each target shape, thereby deforming the basic mesh.

[0087] Step S105: for each target shape, an initial weight value corresponding to the target shape is set according to the phoneme set.

[0088] In step S105 of some embodiments, for each target shape, an initial weight value corresponding to the target shape is set according to the phoneme set. The initial weight value of the phoneme for each target shape refers to the preset weight value of different phonemes for each target shape, which facilitates the subsequent generation of the mouth shape corresponding to the phoneme.

[0089] Step S106, performing weight correction on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data.

[0090] In step S106 of some embodiments, the initial weight value of the target shape is corrected according to the volume annotation file, and the animation weight data is determined by combining the volume annotation file and the phoneme annotation file. This can simultaneously consider the impact of phoneme information and volume changes on the naturalness of the lip shape, so that the lip shape matches the voice emotion expression, thereby improving the accuracy of the lip shape and voice emotion expression.

[0091] Step S107, generating a lip shape corresponding to the character model according to the animation weight data.

[0092] In step S107 of some embodiments, the lip shape corresponding to the character model is generated according to the animation weight data. Specifically, the calculated animation weight data is applied to a predefined fusion deformation set to drive the facial deformation of the 3D character model, achieve synchronous and realistic display of the lip shape and voice, and achieve smooth transition through weight interpolation to enhance the naturalness of the animation.

[0093] Reference Figure 2 , Figure 2 It is a specific flow chart of performing phoneme conversion on the original dialogue text provided by the embodiment of the present application. In some embodiments, the method includes but is not limited to steps S201 to S204.

[0094] Step S201, determining the text category of the original dialogue text, and determining a text processing dictionary according to the text category.

[0095] Step S202, performing word recognition on the original dialogue text through a preset associated word dictionary to obtain monophonetic words and polyphonetic words.

[0096] Step S203, performing pinyin search on monophonetic words and polyphonetic words by using a preset search algorithm to determine the first phoneme text corresponding to the monophonetic word and the second phoneme text corresponding to the polyphonetic word in the text processing dictionary.

[0097] Step S204, generating a dialogue phoneme text according to the first phoneme text and the second phoneme text.

[0098] In steps S201 to S204 of some embodiments, in the process of converting the original dialogue text into phonemes, the embodiments of the present application first determine the text category of the original dialogue text, for example, the original dialogue text is Chinese dialogue text or English dialogue text, etc. Since different language processing may be different, the embodiments of the present application will also determine the text processing dictionary according to the text category. For example, for the original dialogue text in Chinese, the text processing dictionary of the embodiments of the present application is a pinyin dictionary; for the original dialogue text in English, the text processing dictionary of the embodiments of the present application is an English dictionary. Then, the original dialogue text is subjected to word recognition through a preset associated word dictionary to obtain monophonetic words and polyphonetic words. Specifically, the pronunciation of the words in the original dialogue text is determined in combination with the context through natural language processing technology. In this embodiment of the present application, the pronunciation of the words can be identified as one or more through the Chinese word segmentation library and the associated word dictionary to obtain monophonetic words and polyphonetic words, thereby achieving accurate distinction between monophonetic words and polyphonetic words, thereby improving the accuracy of phoneme conversion and ensuring the correct pronunciation of polyphonetic characters. Afterwards, the monophonetic words are subjected to pinyin retrieval through a preset retrieval algorithm to determine the first phoneme text corresponding to the monophonetic word and the second phoneme text corresponding to the polyphonetic word in the text processing dictionary, thereby achieving rapid search of the first phoneme text corresponding to the monophonetic word and the second phoneme text corresponding to the polyphonetic word, improving conversion efficiency, and then generating a dialogue phoneme text based on the first phoneme text and the second phoneme text, so that the dialogue phoneme text corresponds to the original dialogue text, generating fluent and natural voice output, being able to process multiple languages ​​and dialects, and providing high-precision phoneme conversion.

[0099] It should be noted that in the process of performing pinyin retrieval on monophonetic words and polyphonetic words through a preset retrieval algorithm, the embodiment of the present application will quickly retrieve the pinyin of each word through a hash table to achieve efficient data retrieval, and then map the text to the index of the array through a hash function to determine the first phoneme text corresponding to the monophonetic word and the second phoneme text corresponding to the polyphonetic word in the text processing dictionary. In the conversion process, the embodiment of the present application will also remove punctuation marks that are not related to pronunciation. Punctuation marks have no effect on pronunciation. Removing them can reduce unnecessary information, making the phoneme text more concise and facilitating subsequent speech synthesis or speech recognition processing.

[0100] Reference Figure 3 , Figure 3 It is a specific flow chart of volume analysis of the original dialogue sound source provided by the embodiment of the present application. In some embodiments, the method includes but is not limited to steps S301 to S303.

[0101] Step S301, determining the overall sampling duration of the original dialogue audio source, and setting the sampling step length according to the overall sampling duration.

[0102] Step S302: audio sampling is performed on the original dialogue sound source according to the sampling step length to obtain a plurality of sampling frames and volume values ​​corresponding to the sampling frames.

[0103] Step S303: taking the sampling frames and the volume values ​​corresponding to the sampling frames as key frames, and performing normalization processing on all the key frames to generate a volume annotation file.

[0104] In steps S301 to S303 of some embodiments, during the process of performing volume analysis on the original dialogue sound source, the embodiments of the present application will analyze the volume changes in the original dialogue sound source, record the volume level according to the timeline, and generate a volume annotation file. Specifically, the overall sampling duration of the original dialogue sound source is first determined, that is, the entire sampling data of the original dialogue sound source is read, and the sampling step length is set according to the overall sampling duration, so that the step length can be dynamically adjusted according to the actual situation of the original dialogue sound source to balance the simulation accuracy and the use of computing resources and improve the simulation efficiency. Then, the audio sampling of the original dialogue sound source is performed according to the sampling step length, and the maximum or minimum volume value of the audio sampling frame within the step length is taken, wherein the maximum or minimum volume value of the sampling frame is the root mean square sample value of all channels of the frame, and multiple sampling frames and volume values ​​corresponding to the sampling frames are obtained. After that, the sampling frames and the volume values ​​corresponding to the sampling frames are used as key frames, and all key frames are normalized. Specifically, the maximum amplitude of the audio signal in the key frame is calculated, and the normalization multiple is determined. Then, according to the calculated normalization coefficient, the amplitude of the audio signal is adjusted to generate a volume annotation file, thereby enhancing the consistency of the audio and further improving the playback quality and consistency of the original dialogue sound source.

[0105] Reference Figure 4 , Figure 4 It is a specific flow chart of performing phoneme fitting on the original dialogue sound source, dialogue phoneme text and phoneme set provided by the embodiment of the present application. In some embodiments, the method includes but is not limited to steps S401 to S406.

[0106] Step S401, annotate the dialogue phoneme text to obtain corpus data.

[0107] Step S402: extract features from the original conversation sound source using a preset speech recognition tool to obtain a feature vector.

[0108] Step S403, input the feature vector and corpus data into a speech recognition tool for model training to obtain a language acoustic model, and construct a pronunciation dictionary corresponding to the language acoustic model based on the phoneme set.

[0109] Step S404, training a speech recognition model in a speech recognition tool according to a language acoustic model and a pronunciation dictionary to obtain a pre-trained speech recognition model;

[0110] Step S405, for each dialogue phoneme in the dialogue phoneme text, obtain the start timestamp and the end timestamp of the dialogue phoneme, and perform maximum likelihood path fitting on the dialogue phoneme through the pre-trained speech recognition model to obtain the phoneme fitting degree.

[0111] Step S406: Generate a phoneme annotation file according to the start timestamp, the end timestamp and the phoneme fit.

[0112] In steps S401 to S406 of some embodiments, in the process of phoneme fitting of the original dialogue sound source, dialogue phoneme text and phoneme set, the embodiment of the present application will perform text annotation on the dialogue phoneme text to obtain corpus data, and then extract features of the original dialogue sound source through a preset speech recognition tool. Specifically, the embodiment of the present application is based on an open source speech recognition tool Kald based on a deep neural network model with an HMM-GMM (Hidden Markov Model l-Gaussian Mixture Model) architecture. i Extract the spectral features, energy distribution and other speech signal feature data from the original dialogue sound source to obtain the feature vector. Then, input the feature vector and corpus data into the speech recognition tool for model training, and train the language acoustic model corresponding to the character dialogue language, so that the language acoustic model can more accurately capture the characteristics of the speech signal, enhance the accuracy of speech recognition, and facilitate the subsequent improvement of the accuracy of lip synchronization. A pronunciation dictionary corresponding to the language acoustic model is constructed according to the phoneme set, so that the mapping of words to phonemes can be achieved through the pronunciation dictionary, and detailed word pronunciation and phonetic annotation can be provided through the pronunciation dictionary, so as to facilitate the subsequent improvement of the naturalness of speech synthesis. Then, the speech recognition model in the speech recognition tool is trained according to the language acoustic model and the pronunciation dictionary, so as to enhance the speech recognition model for speech recognition. The accuracy of other methods is improved, and the acoustic model and the language model are connected through the pronunciation dictionary to obtain a pre-trained speech recognition model to further improve the accuracy of speech recognition. For each dialogue phoneme in the dialogue phoneme text, the start timestamp and the end timestamp of the dialogue phoneme are obtained, and the dialogue phoneme is fitted with the maximum likelihood path through the pre-trained speech recognition model. The final probability value of the maximum likelihood path in the solution process is used as the fit of the phoneme alignment, that is, the phoneme fit, so that the degree of fit corresponding to each dialogue phoneme can be obtained, and then a phoneme annotation file is generated according to the start timestamp, the end timestamp and the phoneme fit, so that the phoneme annotation file includes the timestamps corresponding to the start and end of each dialogue phoneme, and also includes the fit corresponding to each dialogue phoneme, so that the inherent dependency of time series data can be handled.

[0113] It can be understood that the speech recognition model in the embodiment of the present application is a hidden Markov-mixture Gaussian model.

[0114] Reference Figure 5 , Figure 5 The method includes but is not limited to steps S501 to S504.

[0115] Step S501 : for each target phoneme in the phoneme annotation file, determine a start normalized volume value and an end normalized volume value of the target phoneme according to the volume annotation file.

[0116] Step S502: determining a volume average value according to the start normalized volume value and the end normalized volume value.

[0117] Step S503: Perform a square root process on the volume mean value to obtain a correction multiplier corresponding to the target phoneme.

[0118] Step S504, determining animation weight data according to the correction multiplier and the initial weight value.

[0119] In steps S501 to S504 of some embodiments, in the process of weight correction of the initial weight value of the target shape according to the volume annotation file, for each target phoneme in the phoneme annotation file, the normalized volume value of the start and end points of each target phoneme is first taken, that is, the start normalized volume value and the end normalized volume value of the target phoneme are determined according to the volume annotation file, and then the volume mean is determined according to the start normalized volume value and the end normalized volume value. After that, the volume mean is subjected to quadratic root processing. Since the normalized volume value is taken in the interval [0,1] , whose square root is also between [0,1], and a correction multiplier corresponding to the target phoneme is obtained, wherein the correction multiplier is used to dynamically adjust the weight value based on the volume, so as to facilitate the subsequent simulation of the subtle differences in the lip shape of the animation at different volumes, and then determine the animation weight data based on the correction multiplier and the initial weight value. The embodiment of the present application determines the animation weight data by combining the volume annotation file and the phoneme annotation file, which can simultaneously consider the influence of phoneme information and volume changes on the naturalness of the lip shape, so that the lip shape matches the voice emotion expression, and improves the accuracy of the lip shape and the voice emotion expression.

[0120] It can be understood that if the correction multiplier is 1, it is equivalent to having no effect on the weight value; the smaller the correction multiplier, the smaller the weight value after correction; if the correction multiplier is 0, it is equivalent to all Blend target weight values ​​being corrected to 0, which is equivalent to the corresponding lip animation being in the original undeformed state (generally speaking, it is the state when there is no speaking).

[0121] Reference Figure 6 , Figure 6 6 is a specific flow chart of a method for generating dialogue lip shapes provided by another embodiment of the present application. In some embodiments, the method includes but is not limited to steps S601 to S606.

[0122] It should be noted that steps S601 to S606 occur after the initial weight value of the target shape is modified according to the volume annotation file to obtain the animation weight data.

[0123] Step S601: merge the data in the volume annotation file and the data in the phoneme annotation file into the same time axis to obtain a target time axis.

[0124] Step S602 , performing interval screening on the target time axis according to a preset normalized volume threshold, so as to screen out target sections that are greater than the normalized volume threshold.

[0125] Step S603 : when the target segment does not cover the phoneme, determine the first segment and the second segment adjacent to the target segment, and calculate the segment normalized volume mean of the target segment.

[0126] Step S604, obtaining a first degree of fit corresponding to the first segment and a second degree of fit corresponding to the second segment.

[0127] Step S605 , determining the segment target value according to the first fit degree, the second fit degree and the segment normalized volume mean.

[0128] Step S606: When the segment target value is less than a preset constant value, the phonemes on the target time axis are aligned and corrected.

[0129] In steps S601 to S606 of some embodiments, after the initial weight value of the target shape is weight-corrected according to the volume annotation file, the embodiments of the present application will also correct the position of the phoneme on the time axis. Specifically, the data in the volume annotation file and the data in the phoneme annotation file are merged into the same time axis to obtain the target time axis, and then the target time axis is interval-filtered according to a preset normalized volume threshold, and all segments whose normalized volume is greater than the threshold are searched on the target time axis to filter out the target segment greater than the normalized volume threshold. Since there may be a situation where the speech recognition model has poor phoneme recognition effect and alignment deviation occurs, the embodiment of the present application will determine whether the target segment covers the phoneme. When the target segment does not cover the phoneme, the first segment and the second segment adjacent to the target segment are determined, wherein the first segment is the previous segment adjacent to the target segment, and the second segment is the next segment adjacent to the target segment, and the segment normalized volume mean of the target segment is calculated, and then the first fit corresponding to the first segment and the second fit corresponding to the second segment are obtained, the first fit is divided by the segment normalized volume mean to obtain a first value, and the second fit is divided by the segment normalized volume mean to obtain a second value, and the segment target value is determined according to the first value and the second value, so as to facilitate the subsequent determination of whether the alignment deviation occurs through the segment target value. When the segment target value is less than the preset constant value, it means that the alignment deviation occurs at this time, and the phonemes on the target time axis need to be aligned and corrected, so as to improve the alignment deviation when the recognition effect of some phonemes is poor, further improve the alignment effect of the sound and the mouth shape, and avoid the situation where there is no corresponding mouth shape in some pronunciation areas.

[0130] It is understandable that the preset constant value in the embodiment of the present application can be set according to the user's needs, for example, set to 1, 2 or 5, etc., and the embodiment of the present application does not make specific limitations.

[0131] Reference Figure 7 , Figure 7 It is a specific flow chart of aligning and correcting phonemes on a target time axis provided by an embodiment of the present application. In some embodiments, the method includes but is not limited to steps S701 to S704.

[0132] Step S701, determining the original phoneme alignment time point according to the phoneme annotation file, wherein the original phoneme alignment time point is the midpoint between the timestamp of the start of the phoneme and the timestamp of the end of the phoneme.

[0133] Step S702: determine the segment center time point of the target segment.

[0134] Step S703, determining the time median of the original phoneme alignment time point and the segment center time point, and using the time median as the offset midpoint.

[0135] Step S704: Correct the position of the phoneme on the target time axis according to the offset midpoint.

[0136] In steps S701 to S704 of some embodiments, in the process of aligning and correcting the phonemes on the target time axis, the embodiments of the present application first determine the original alignment time point of the phoneme based on the phoneme annotation file, wherein the original alignment time point of the phoneme is the midpoint from the timestamp at the beginning of the phoneme to the timestamp at the end of the phoneme, and then determine the segment center time point of the target segment, and then determine the time median of the original alignment time point of the phoneme and the segment center time point, and use the time median as the offset midpoint, and then offset the time point corresponding to the phoneme on the target time axis to the offset midpoint, thereby achieving the correction alignment of the phonemes on the target time axis and avoiding the occurrence of alignment deviation.

[0137] See also Figure 8 The embodiment of the present application also provides a dialogue lip shape generation system, which can implement the above-mentioned dialogue lip shape generation method, and the system includes:

[0138] The data acquisition module 801 is used to acquire the original dialogue text, the original dialogue sound source and the phoneme set corresponding to the type of the original dialogue text;

[0139] The data analysis module 802 is used to perform phoneme conversion on the original dialogue text to obtain a dialogue phoneme text, and perform volume analysis on the original dialogue sound source to generate a volume annotation file;

[0140] The phoneme fitting module 803 is used to perform phoneme fitting on the original dialogue sound source, dialogue phoneme text and phoneme set to generate a phoneme annotation file;

[0141] The model acquisition module 804 is used to acquire a character model in a preset fusion deformation set and acquire multiple target shapes corresponding to the character model;

[0142] A weight setting module 805, for setting, for each target shape, an initial weight value corresponding to the target shape according to the phoneme set;

[0143] The weight correction module 806 is used to perform weight correction on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data;

[0144] The lip shape generation module 807 is used to generate the lip shape corresponding to the character model according to the animation weight data.

[0145] The dialogue lip shape generation system of the embodiment of the present application is used to execute the dialogue lip shape generation method in the above embodiment. Its specific processing process is the same as the dialogue lip shape generation method in the above embodiment, and will not be repeated here.

[0146] In order to more clearly explain the dialogue lip shape generation method, system and electronic device of the embodiment of the present application, specific examples are given below for illustration.

[0147] Example 1:

[0148] See also Fig. 9 , Fig. 9 It is a schematic diagram of the overall framework provided by an example of this application.

[0149] The embodiment of the present application is applicable to the automatic generation of lip shapes in different languages. Since different languages ​​have great differences in phoneme recognition and lip shape correspondence, in order to achieve a better lip shape generation effect, the embodiment of the present application defines an independent phoneme set for each different language. For example, for the Chinese phonetic language, we define the phoneme set as the set of all pinyin initials and all pinyin finals.

[0150] First, the prepared original dialogue text is converted into dialogue phoneme text. The conversion process is based on word pronunciation recognition in various languages. Different languages ​​may be processed differently. For example, for Chinese, we need a Chinese character pinyin dictionary, and add related words to polyphonic characters to assist inference, and then use common retrieval algorithms such as hash tables for retrieval. During the conversion process, we remove punctuation marks that are not related to pronunciation, and finally output the dialogue phoneme text corresponding to the dialogue text.

[0151] After that, a phoneme annotation file is generated based on the original dialogue sound source and the dialogue phoneme text. The open source speech recognition tool Kald i, which is based on the deep neural network model of the HMM-GMM architecture, is used to extract the spectral features, energy distribution and other speech signal feature data in the original dialogue audio to obtain the feature vector. Then, the original dialogue text is annotated to obtain the corpus data. The prepared corpus data and feature vectors are used to train the language acoustic model corresponding to the character dialogue language through the Kald i tool, and the pronunciation dictionary corresponding to the acoustic model is constructed according to the predefined phoneme set. Then, for each phoneme in the dialogue phoneme text, the HMM-GMM model in the Kald i tool is used to perform the maximum likelihood path fitting solution, and the final probability value of the maximum likelihood path in the solution process is used as the fit of the phoneme alignment, and finally a phoneme annotation file containing the start and end timestamps and fit of each phoneme is generated.

[0152] It should be noted that by extracting the HMM-GMM model to solve the probability value of the maximum likelihood path as the fit of the phoneme alignment, and then correcting the alignment result with low fit through volume annotation data, the situation where the pronunciation area is missing and the corresponding lip shape is missing when the phoneme recognition effect is poor can be significantly improved.

[0153] Then, the volume of the original dialogue sound source is analyzed, the volume changes in the original dialogue sound source are analyzed, the volume level is recorded according to the time axis, and a volume annotation file is generated. Specifically, the volume annotation file is generated by reading the entire sampling data of the audio, setting a step length, taking the maximum / minimum volume value of the audio sampling frame within the step length (the volume value of the sampling frame is calculated as: the root mean square sample value of all channels of the frame), recording the corresponding frame number and volume value as the key frame, extracting all the key frames, and normalizing the volume values ​​of all the key frames to obtain the volume annotation file.

[0154] In the binding of phonemes and lip animation, the embodiment of the present application adopts animation generation technologies such as skeletal animation and BlendShape. Taking BlendShape as an example, the embodiment of the present application creates a BlendShape target for each different character lip shape. For example, different targets can be created for lips closed, lips open, corners of the mouth raised, etc. Then, the initial weight value of each phoneme in the phoneme set for each BlendShape target needs to be set.

[0155] The weight value is not only based on the phoneme type, but also dynamically adjusted according to the volume to simulate the subtle differences in the speaker's mouth shape at different volumes. The algorithm for correcting the weight value by volume is as follows: take the normalized volume value of the start and end points of the phoneme, take the average, take the square root of the value as the correction multiplier of the BlendShape weight value, and finally multiply the correction multiplier by the initial weight value to obtain the animation weight data.

[0156] When the animation weight data is mixed and calculated, the embodiment of the present application will also use the phoneme annotation data as a benchmark, and correct the time axis mark of the phoneme according to the volume annotation data. The specific algorithm is: set a normalized volume threshold, such as 0.5. Then merge the phoneme annotation data and the volume annotation data onto the same time axis. Afterwards, search for all segments on the time axis whose normalized volume is greater than the threshold. If there is no phoneme coverage in the eligible segment, search for the phonemes before and after the segment, and divide the fit of the phonemes before and after by the mean of the normalized volume of the segment. If the value is less than a preset constant value (for example, 1), perform phoneme alignment correction. Specifically, the time point corresponding to the phoneme is offset to the center of the segment, and the offset point is the median of the original alignment time point of the phoneme and the time point of the center of the segment. This correction algorithm can be executed multiple times, which can significantly improve the alignment deviation when the recognition effect of some phonemes is not good, resulting in the situation where there is no corresponding lip shape in some pronunciation areas.

[0157] Finally, the calculated animation weight data is applied to the predefined BlendShape set to drive the facial deformation of the 3D character model and achieve a synchronized and realistic display of lip shape and speech. The preset BlendShape should cover common lip shape changes and achieve smooth transitions through weight interpolation to enhance the naturalness of the animation.

[0158] In some embodiments, the embodiments of the present application match lip animation based on phoneme alignment. Compared with most solutions on the market that use direct speech recognition, there will be no misrecognition of phonemes, and the accuracy of phoneme alignment is also better than direct speech recognition. In addition, the embodiments of the present application define different phoneme sets for different languages, and define the character's lip shape according to different phoneme sets, which can generate the character's lip shape effects for dialogues in different languages ​​more finely and accurately. In addition, the use of volume annotation data mixed to calculate animation weights can better present the subtle differences in the speaker's lip shape at different volumes, and can make the transition of the character's dialogue lip shape appear more natural and smooth.

[0159] The system generates accurate phoneme text according to the content of the dialogue, uses the speech recognition engine to align the phonemes, and then uses the volume annotation data for correction to generate the character lip animation. Compared with many single solutions on the market, it can achieve higher accuracy in matching pronunciation and dialogue lip shape, and does not rely too much on the accuracy of the speech recognition model in terms of fault tolerance. In terms of generation effect, the lip animation weights that integrate the volume annotation data make the dialogue lip transition more refined and natural.

[0160] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute the dialogue lip shape generation method in the above embodiment of the present application.

[0161] See also Fig.10 , Fig.10 The hardware structure of the electronic device provided in the embodiment of the present application is illustrated, and the electronic device includes:

[0162] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0163] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the dialogue lip generation method of the embodiment of this application;

[0164] Input / output interface 903, used to implement information input and output;

[0165] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0166] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0167] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0168] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned dialogue lip shape generation method is implemented.

[0169] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0171] It can be understood by those skilled in the art that Figure 1-10 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0172] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0173] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0174] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0175] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0176] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0177] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0179] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (RandomAccess Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0180] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating dialogue lip shapes, characterized in that: The method comprises: Acquire an original dialogue text, an original dialogue sound source, and a phoneme set corresponding to the type of the original dialogue text; Performing phoneme conversion on the original dialogue text to obtain a dialogue phoneme text, and performing volume analysis on the original dialogue sound source to generate a volume annotation file; Performing phoneme fitting on the original dialogue sound source, the dialogue phoneme text and the phoneme set to generate a phoneme annotation file; Obtaining a character model in a preset fusion deformation set, and obtaining a plurality of target shapes corresponding to the character model; For each of the target shapes, setting an initial weight value corresponding to the target shape according to the phoneme set; Performing weight correction on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data; A lip shape corresponding to the character model is generated according to the animation weight data.

2. The method for generating dialogue lip shapes according to claim 1, characterized in that: The step of converting the original dialogue text into phonemes to obtain a dialogue phoneme text comprises: Determining a text category of the original conversation text, and determining a text processing dictionary according to the text category; Performing word recognition on the original dialogue text through a preset associated word dictionary to obtain monophonetic words and polyphonetic words; Performing pinyin search on the monophonetic word and the polyphonetic word by using a preset search algorithm to determine a first phoneme text corresponding to the monophonetic word and a second phoneme text corresponding to the polyphonetic word in the text processing dictionary; A dialogue phoneme text is generated according to the first phoneme text and the second phoneme text.

3. The method for generating dialogue lip shapes according to claim 1, characterized in that: The step of performing volume analysis on the original dialogue sound source to generate a volume annotation file includes: Determining the overall sampling duration of the original dialogue audio source, and setting the sampling step length according to the overall sampling duration; Performing audio sampling on the original dialogue sound source according to the sampling step length to obtain a plurality of sampling frames and volume values ​​corresponding to the sampling frames; The sampling frames and the volume values ​​corresponding to the sampling frames are used as key frames, and all the key frames are normalized to generate a volume annotation file.

4. The method for generating dialogue lip shapes according to claim 1, characterized in that: The performing phoneme fitting on the original dialogue sound source, the dialogue phoneme text and the phoneme set to generate a phoneme annotation file includes: Performing text annotation on the dialogue phoneme text to obtain corpus data; Extracting features from the original conversation sound source using a preset speech recognition tool to obtain a feature vector; Inputting the feature vector and the corpus data into the speech recognition tool for model training to obtain a language acoustic model, and constructing a pronunciation dictionary corresponding to the language acoustic model according to the phoneme set; Training the speech recognition model in the speech recognition tool according to the language acoustic model and the pronunciation dictionary to obtain a pre-trained speech recognition model; For each dialogue phoneme in the dialogue phoneme text, obtaining a start timestamp and an end timestamp of the dialogue phoneme, and performing a maximum likelihood path fitting solution on the dialogue phoneme through the pre-trained speech recognition model to obtain a phoneme fitting degree; A phoneme annotation file is generated according to the start timestamp, the end timestamp and the phoneme fit.

5. The method for generating dialogue lip shapes according to claim 1, characterized in that: The step of performing weight correction on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data includes: For each target phoneme in the phoneme annotation file, determining a start normalized volume value and an end normalized volume value of the target phoneme according to the volume annotation file; Determining a volume mean value according to the start normalized volume value and the end normalized volume value; Performing a square root process on the volume mean to obtain a correction multiplier corresponding to the target phoneme; Animation weight data is determined according to the correction multiplier and the initial weight value.

6. The method for generating dialogue lip shapes according to claim 1, characterized in that: After weight correction is performed on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data, the method further includes: Merging the data in the volume annotation file and the data in the phoneme annotation file into the same time axis to obtain a target time axis; Performing interval screening on the target time axis according to a preset normalized volume threshold to screen out a target segment greater than the normalized volume threshold; When the target segment does not cover the phoneme, determining a first segment and a second segment adjacent to the target segment, and calculating a segment normalized volume mean of the target segment; Obtaining a first degree of fit corresponding to the first segment and a second degree of fit corresponding to the second segment; Determining a segment target value according to the first fit degree, the second fit degree, and the segment normalized volume mean; When the segment target value is less than a preset constant value, the phonemes on the target time axis are aligned and corrected.

7. The method for generating dialogue lip shapes according to claim 6, characterized in that: The aligning and correcting the phonemes on the target time axis includes: Determine the phoneme original alignment time point according to the phoneme annotation file, wherein the phoneme original alignment time point is the midpoint between the timestamp of the beginning of the phoneme and the timestamp of the ending of the phoneme; Determining a segment center time point of the target segment; Determine the time median of the phoneme original alignment time point and the segment center time point, and use the time median as the offset midpoint; The position of the phoneme on the target time axis is corrected according to the offset midpoint.

8. A dialogue lip-sync generation system, characterized in that: The system comprises: A data acquisition module, used to acquire the original dialogue text, the original dialogue sound source and a phoneme set corresponding to the type of the original dialogue text; A data analysis module, used to perform phoneme conversion on the original dialogue text to obtain a dialogue phoneme text, and perform volume analysis on the original dialogue sound source to generate a volume annotation file; A phoneme fitting module, used for performing phoneme fitting on the original dialogue sound source, the dialogue phoneme text and the phoneme set to generate a phoneme annotation file; A model acquisition module, used to acquire a character model in a preset fusion deformation set, and acquire multiple target shapes corresponding to the character model; A weight setting module, configured to set, for each of the target shapes, an initial weight value corresponding to the target shape according to the phoneme set; A weight correction module, used for performing weight correction on the initial weight value of the target shape according to the volume annotation file to obtain animation weight data; A lip shape generation module is used to generate a lip shape corresponding to the character model according to the animation weight data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method for generating dialogue lip shapes according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the dialogue lip-sync generation method according to any one of claims 1 to 7.

Images