Mouth shape synchronization method and system for 3D digital human

Abstract

The invention discloses a mouth shape synchronization method and system for a 3D digital human. The method comprises the following steps: screening an audio file; carrying out ASR identification on the audio file to obtain a text expression corresponding to the audio file; performing alignment operation on the audio file and the corresponding text expression by using a multi-modal phoneme alignment tool to generate a phoneme time sequence; further generating a mouth shape animation time sequence according to the phoneme time sequence; classifying the Chinese phoneme mouth shapes, and creating BlendShape for various mouth shapes according to a classification result; carrying out weighted adjustment on the BlendShape weight of the corresponding mouth shape; and based on the mouth shape BlendShape after weight adjustment, using Unity to carry out modeling on a 3D task, and establishing a corresponding image model for each mouth shape. The method has the advantages that the vivid degree of the 3D digital human is remarkably improved, the mouth shape and voice synchronization of the digital human is more accurate and natural, and the immersion and substitution of audiences are enhanced.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of computer graphics and speech processing, and in particular to a lip synchronization method and system for 3D digital humans. Background Art

[0002] Some existing lip synchronization technologies are not accurate and natural enough in matching lip shape with speech. There are problems such as the lip shape changes being too abrupt and the lip shape and speech rhythm being out of sync, which is not very harmonious visually. For complex Chinese phoneme combinations and changeable speech features, existing technologies find it difficult to accurately generate corresponding lip shapes. Summary of the invention

[0003] The object of the present invention is to provide a lip synchronization method and system for 3D digital human, so as to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A lip synchronization method for 3D digital human, comprising the following steps:

[0006] S1. Audio data collection: Collect Chinese human voice audio, and manually proofread the collected audio to select audio files with standard pronunciation and clarity;

[0007] S2, audio to text conversion: use the acoustic model and language model to process the audio file and the output of the acoustic model respectively; use the decoding algorithm to integrate the output of the language model and the acoustic model to obtain the text expression corresponding to the audio file;

[0008] S3. Phoneme alignment: Use multimodal phoneme alignment tools to align audio files and their corresponding text expressions to generate phoneme time series; predict the corresponding lip state at each moment based on the phoneme time series to generate a lip animation time series;

[0009] S4, mapping of phonemes to visemes: Based on the mapping relationship between the phoneme time series and the lip shape animation time series, the Chinese phoneme lip shapes are classified, and BlendShape is created for each type of lip shape according to the classification results. The BlendShape weight is set to represent the lip shapes of different phonemes, thus realizing the mapping of phonemes to lip shapes.

[0010] S5. BlendShape weight calculation of mouth shape: based on the speech feature parameters of the audio, the BlendShape weight of the corresponding mouth shape is adjusted, the adjusted BlendShape weight is saved in the form of key frames and the number of key frames is optimized to a certain extent;

[0011] S6. Unity modeling: Post-production artists and animation workers use Unity to model 3D tasks based on the weighted lip shape BlendShape and establish corresponding image models for each lip shape.

[0012] Preferably, step S2 specifically includes the following contents:

[0013] S21, collecting audio signals in audio files and preprocessing them;

[0014] S22, extracting key acoustic features from the preprocessed audio signal;

[0015] S23, analyzing and modeling the extracted acoustic features using an acoustic model to predict speech units that may be contained in the audio;

[0016] S24, using the language model to constrain and optimize the prediction results of the acoustic model based on the grammar, semantics and statistical laws of the language;

[0017] S25. Based on the output of the decoding algorithm integrated acoustic model and language model, the predicted speech unit is converted into complete and accurate text content, and the text expression corresponding to the audio file is generated by searching for the optimal path or combination; the text expression adopts the pinyin form with numbers representing the tones.

[0018] Preferably, step S3 specifically includes the following contents:

[0019] S31, analyzing and extracting features from the audio signal in the audio file to obtain key features of the audio;

[0020] S32, parsing and encoding the text expression corresponding to the audio file, converting each pinyin character of the text expression and its corresponding tone number into a format that is convenient for calculation and comparison;

[0021] S33, using a multimodal phoneme alignment tool to find the best correspondence between the key features of the audio and the text expression after the conversion format, and gradually find the optimal matching path through dynamic planning, so as to determine the exact position of each phoneme on the time axis and generate a phoneme time series;

[0022] S34. Model the lip movements according to the characteristics and changing rules of the phonemes. Based on the generated phoneme time series, use the constructed model to predict the lip state corresponding to each moment, and generate a time series of lip animation.

[0023] Preferably, step S4 specifically includes the following contents:

[0024] S41, based on the mapping relationship between the phoneme time series and the lip shape animation time series, classifying the lip shapes of the phonemes according to the pronunciation characteristics and visual similarity of various Chinese phonemes;

[0025] S42, based on the mouth shape classification result of the phonemes, creating a mouth shape model of a specific mouth shape for each classification, that is, a specific type of BlendShape;

[0026] S43. Set the weight of BlendShape to represent the mouth shapes corresponding to different phonemes.

[0027] Preferably, step S42 specifically includes: analyzing the common lip shape characteristics of the phonemes contained in each category, using 3D modeling software or animation production tools to create an initial basic lip shape model; adjusting and modifying the basic lip shape model based on the characteristics of each category to make it conform to the typical lip shape of the phonemes in the corresponding category; through multiple adjustments and optimizations, until a model that can accurately express the lip shape characteristics of the corresponding category phonemes, that is, a specific type of BlendShape, is created.

[0028] Preferably, step S5 specifically includes the following contents:

[0029] S51, sampling the audio signal at a fixed sampling frequency according to the phoneme sequence, and calculating the speech feature parameters of the sampled audio;

[0030] S52, according to the voice feature parameters, weighted adjustment is performed on the BlendShape weight of the corresponding mouth shape;

[0031] S53, saving the weighted BlendShape weight after weighted adjustment as an animation curve in the form of key frames;

[0032] S54. Optimize the number of key frames and reduce unnecessary key frames.

[0033] Preferably, step S52 specifically includes the following contents:

[0034] S521, setting corresponding judgment thresholds and ranges for speech feature parameters related to the lip shape, including three ranges of high, medium and low;

[0035] S522. Compare the sampled speech feature parameters with the judgment threshold of the corresponding lip shape classification to determine the range of the speech feature parameters. If it is in a low range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameter is slightly increased or remains unchanged; if it is in a medium range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameter is appropriately increased; if it is in a high range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameter is greatly increased.

[0036] Preferably, the voice feature parameters include volume, pitch, and speaking speed.

[0037] Preferably, when increasing the weight, the specific increase is calculated according to a pre-set rule or algorithm; in the BlendShape weight adjustment process, the comprehensive influence of multiple speech feature parameters should also be considered, and the adjustment results of multiple parameters are combined by weighted averaging and priority sorting to finally determine the specific adjustment value of the BlendShape weight for each lip shape.

[0038] The present invention also aims to provide a lip synchronization system for 3D digital human, which can implement the above-mentioned method, and comprises:

[0039] Audio data collection module: collects Chinese human voice audio, and manually proofreads the collected audio to select audio files with standard pronunciation and clarity;

[0040] Audio-to-text module: Use the acoustic model and language model to process the audio file and the output of the acoustic model respectively; use the decoding algorithm to combine the output of the language model and the acoustic model to obtain the text expression corresponding to the audio file;

[0041] Phoneme alignment module: Use multimodal phoneme alignment tools to align audio files and their corresponding text expressions to generate phoneme time series; predict the corresponding lip state at each moment based on the phoneme time series to generate a lip animation time series;

[0042] Phoneme to viseme mapping module: Based on the mapping relationship between the phoneme time series and the lip shape animation time series, the Chinese phoneme lip shapes are classified, and BlendShape is created for each type of lip shape according to the classification results. The BlendShape weight is set to represent the lip shapes of different phonemes, thus realizing the mapping from phonemes to lip shapes.

[0043] Lip shape BlendShape weight calculation module: Based on the voice feature parameters of the audio, the BlendShape weight of the corresponding lip shape is adjusted, the adjusted BlendShape weight is saved in the form of key frames and the number of key frames is optimized to a certain extent;

[0044] Unity modeling module: Post-production artists and animators use Unity to model 3D tasks based on the weighted lip shape BlendShape, and establish corresponding image models for each lip shape.

[0045] The beneficial effects of the present invention are: 1. The realism of 3D digital human is significantly improved, and the synchronization of digital human's lip shape and voice is more accurate and natural, which enhances the audience's immersion and sense of substitution. 2. The lip shape of 3D digital human is highly matched with Chinese voice, which enhances the visual sense of reality and fluency, and can effectively handle the combination and changes of various Chinese phonemes to adapt to different voice situations. 3. The pronunciation lip shape of various Chinese phonemes is accurately presented, and the synchronization effect is also improved, which helps to convey voice information more clearly, so that it can be more widely used in virtual anchors, online meetings, games and other fields, providing a better interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a lip synchronization method in an embodiment of the present invention;

[0047] Figure 2 is a flowchart of audio-to-text conversion in an embodiment of the present invention;

[0048] Figure 3 is a flowchart of phoneme alignment in an embodiment of the present invention;

[0049] Figure 4 It is a flow chart of the lip shape BlendShape weight calculation in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] like Figure 1 As shown, in this embodiment, a lip synchronization method for 3D digital human is provided, so that the lip shape of 3D digital human is highly matched with Chinese speech, the visual sense of reality and fluency is enhanced, and the combination and change of various Chinese phonemes can be effectively processed to adapt to different speech situations, thereby solving the problems existing in the prior art. This method includes the following parts:

[0052] 1. Audio Data Collection

[0053] Collect Chinese human voice audio, manually proofread the collected audio, and select audio files with standard pronunciation and clarity.

[0054] 2. Audio to Text

[0055] Automatic speech recognition (ASR) is used to process audio files in order to obtain the accurate text content corresponding to the audio files. That is, the audio files and the output of the acoustic models are processed using the acoustic model and the language model respectively; the output of the language model and the acoustic model are combined using the decoding algorithm to obtain the text expression corresponding to the audio files. Figure 2 As shown, the specific identification process is:

[0056] 2.1. Collect audio signals from audio files and preprocess them, including noise removal, gain adjustment and other operations to improve the quality and recognizability of the audio.

[0057] 2.2. Extract key acoustic features, such as spectrum, cepstrum, etc., from the preprocessed audio signal.

[0058] 2.3. Use acoustic models (such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), etc.) to analyze and model the extracted acoustic features to predict the speech units that may be contained in the audio.

[0059] 2.4. The language model is used to constrain and optimize the prediction results of the acoustic model based on the grammar, semantics and statistical laws of the language. It takes into account the frequency of occurrence of words, contextual relationships, etc. to improve the accuracy and rationality of recognition. For example, if the acoustic model predicts a less common word combination, the language model may make corrections based on language habits to improve the accuracy and rationality of recognition.

[0060] 2.5. Based on the output of the comprehensive acoustic model and language model, the decoding algorithm converts the predicted speech unit into complete and accurate text content, and generates the text expression corresponding to the audio file by searching for the optimal path or combination.

[0061] 3. Phoneme Alignment

[0062] Using the Multimodal Phoneme Alignment Tool (MFA), the audio file and its corresponding text expression (the text expression uses a pinyin form with numbers representing the pitch) are precisely aligned to generate the corresponding phoneme time series; based on the phoneme time series, the corresponding lip state at each moment is predicted to generate a lip animation time series. Figure 3 As shown, it specifically includes the following contents:

[0063] 3.1. Analyze and extract features of audio signals in audio files, and obtain key features of audio such as frequency, duration, intensity, etc. from the audio through relevant technologies.

[0064] 3.2. Parse and encode the textual expression corresponding to the audio file, and convert each pinyin character and its corresponding tone number in the textual expression into a format that is easy to calculate and compare. For example, "ā" will be encoded as a specific identifier.

[0065] 3.3. Use multimodal phoneme alignment tools to find the best correspondence between the key features of the audio and the text expression after conversion, and gradually find the optimal matching path through dynamic planning, so as to determine the exact position of each phoneme on the time axis and generate a phoneme time sequence. For example, determine the start and end time of the sound "a" in the audio and generate a phoneme time sequence.

[0066] 3.4. Use specific algorithms and models to model lip movements based on the characteristics and change patterns of phonemes. Consider the impact of phoneme pronunciation on lips, tongue, etc., and parameterize the changes in lip shape. Based on the generated phoneme time series, use the constructed model to predict the corresponding lip state at each moment and generate a time series of lip shape animation. For example, for the "o" sound, a lip shape animation time point with round lips will be generated.

[0067] 4. Mapping of Phonemes to Visemes

[0068] Based on the mapping relationship between the phoneme time series and the lip shape animation time series, the Chinese phoneme lip shapes are classified. According to the classification results, a BlendShape of a certain category of lip shapes can be created, and the lip shapes of different phonemes can be represented by setting the weight of the BlendShape, thereby realizing the mapping of phonemes to lip shapes. The specific contents include the following:

[0069] 4.1. Based on the mapping relationship between the phoneme time series and the lip animation time series, the phoneme mouth shapes are classified according to the pronunciation characteristics of various Chinese phonemes (such as lip shape, tongue position, etc.) and visual similarity. For example, a, an, uang, etc. can be classified as mth-a mouth shape, o, u, ou, etc. can be classified as mth-o, b, p, m, etc. can be classified as mth-b, zh, ch, sh, etc. can be classified as mth-r, and so on.

[0070] 4.2. Based on the results of the phoneme lip shape classification, a lip shape model with a specific lip shape is created for each classification, that is, a specific type of BlendShape. The specific creation process is as follows:

[0071] First, analyze the common mouth shape characteristics of the phonemes contained in each category. For example, for the category "mth-a", the common characteristics may be a wide mouth opening and a naturally stretched lip state. Then, use 3D modeling software or animation tools to create an initial basic mouth shape model. Next, adjust and modify this basic model based on the characteristics of each category. For example, for the category "mth-a", the mouth opening may be further expanded, the shape of the lips may be adjusted, etc., so that it conforms to the typical mouth shape of the phonemes in this category. Through multiple adjustments and optimizations, until a model that can accurately express the mouth shape characteristics of the phonemes in this category is created, that is, a specific type of BlendShape. BlendShape is a technology used to control the shape change of a model in 3D animation. It can achieve the transition of various intermediate shapes by setting different weights. In this way, when it is necessary to express the mouth shape of a certain phoneme, the BlendShape of the corresponding category can be called, and the precise mouth shape change can be achieved according to the set weight.

[0072] 4.3. Set the weight of BlendShape to indicate the mouth shape corresponding to different phonemes. The weight determines the influence and proportion of each phoneme on the mouth shape. By adjusting the weight, the change of the mouth shape can be accurately controlled, thus realizing the mapping of phonemes to mouth shapes.

[0073] 5. Mouth shape BlendShape weight calculation

[0074] Based on the speech feature parameters of the audio, the BlendShape weights of the corresponding lip shapes in the fourth part are adjusted weightedly, thereby affecting the degree of opening and closing of the lip shapes. The calculated BlendShape weights are saved as animation curves in the form of keyframes and the number of keyframes is optimized to provide editing and adjustment for later artists and animators. Weight adjustment can make the generated lip animation more natural and realistic, and provide an editable and adjustable basis for later artists and animators to meet higher creative needs. Figure 4 As shown, it specifically includes the following contents:

[0075] 5.1. Sample the audio signal at a fixed sampling frequency according to the phoneme sequence, and calculate the speech feature parameters (including volume, pitch, and speech speed) of the sampled audio.

[0076] 5.2. According to the speech feature parameters, the weight of the BlendShape of the corresponding mouth shape is adjusted; for example, if the audio volume is high, the weight corresponding to the opening and closing degree of certain mouth shapes may be increased. The specific adjustment process is:

[0077] (1) First, determine the threshold and range of the speech feature parameters related to the mouth shape. For example, for the audio volume, set the low, medium, and high ranges of the volume and their specific judgment thresholds.

[0078] (2) Then, the sampled speech feature parameters are compared with the judgment threshold of the corresponding lip shape classification to determine the range of the speech feature parameters. If they are in the low range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameters is slightly increased or remains unchanged; if they are in the medium range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameters is appropriately increased; if they are in the high range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameters is greatly increased.

[0079] For example, if the volume is in a low range, the weight corresponding to the degree of lip opening and closing may be slightly increased or remain unchanged; if the volume is in a medium range, the weight is increased appropriately; and if the volume is in a high range, the weight is increased significantly.

[0080] (3) When increasing the weight, the specific increase is calculated according to a pre-set rule or algorithm. This may be based on a linear relationship, a nonlinear relationship, or a more complex function. For example, assuming that the volume and the increase in weight are linearly related, the weight increases by a fixed value for each unit increase in volume.

[0081] (4) A similar approach is also used for other speech feature parameters, such as pitch, speech rate, etc. According to their specific values ​​and the set threshold range, the BlendShape weight of the corresponding mouth shape is adjusted according to the corresponding rules.

[0082] (5) At the same time, the comprehensive influence of multiple speech feature parameters must also be considered. The adjustment results of multiple parameters may be combined through weighted average, priority sorting, etc., and the specific adjustment value of each lip shape BlendShape weight may be finally determined.

[0083] 5.3. Save the weighted BlendShape weights after weighted adjustment as animation curves in the form of keyframes; keyframes record the weight values ​​at specific moments, and continuous lip shape change animations can be generated through these keyframes.

[0084] 5.4. Optimize the number of key frames and reduce unnecessary key frames to improve data processing efficiency and animation fluency.

[0085] In this embodiment, during the BlendShape weight adjustment process, the comprehensive influence of multiple speech feature parameters should be considered, and the adjustment results of multiple parameters should be combined by weighted averaging and priority sorting to ultimately determine the specific adjustment value of the BlendShape weight for each lip shape.

[0086] 6. Unity Modeling

[0087] Post-production artists and animators use Unity to model 3D tasks based on the weighted lip shape BlendShape and create corresponding image models for each lip shape.

[0088] Unity can create a corresponding image model for each mouth shape, in which the facial features of the character, including the shape and structure of the lips, teeth, tongue and other parts, are shaped very carefully to ensure that each mouth shape can be accurately and vividly presented on the 3D model. For example, for the mouth shape corresponding to the phoneme "a", an image model with a wide open mouth may be created, and the shape and stretch of the lips are carefully designed; for the mouth shape corresponding to the phoneme "o", a model with round and slightly protruding lips may be created.

[0089] In this embodiment, a lip synchronization system for 3D digital human is provided. The system can implement the above-mentioned method. The system includes:

[0090] (1) Audio data collection module: collects Chinese human voice audio, and manually proofreads the collected audio to select audio files with standard pronunciation and clarity.

[0091] (2) Audio-to-text module: The acoustic model and language model are used to process the audio file and the output of the acoustic model respectively; the decoding algorithm is used to integrate the output of the language model and the acoustic model to obtain the text expression corresponding to the audio file.

[0092] (3) Phoneme alignment module: Use the multimodal phoneme alignment tool to align the audio file and its corresponding text expression to generate a phoneme time series; predict the corresponding lip state at each moment based on the phoneme time series to generate a lip animation time series.

[0093] (4) Phoneme to viseme mapping module: Based on the mapping relationship between the phoneme time series and the lip shape animation time series, the Chinese phoneme lip shapes are classified. BlendShape is created for each type of lip shape according to the classification results. The BlendShape weight is set to represent the lip shapes of different phonemes, thus realizing the mapping of phonemes to lip shapes.

[0094] (5) Lip shape BlendShape weight calculation module: Based on the speech feature parameters of the audio, the BlendShape weight of the corresponding lip shape is weighted and adjusted.

[0095] (6) Unity modeling module: Post-production artists and animation workers use Unity to model 3D tasks based on the weighted lip shape BlendShape and create corresponding image models for each lip shape.

[0096] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0097] The present invention provides a lip synchronization method and system for 3D digital human, which significantly improves the realism of 3D digital human, makes the synchronization of digital human's lip shape and voice more accurate and natural, and enhances the audience's immersion and sense of substitution. The lip shape of 3D digital human is highly matched with Chinese voice, which enhances the visual sense of reality and fluency, and can effectively handle the combination and change of various Chinese phonemes to adapt to different voice situations. The pronunciation lip shape of various Chinese phonemes is accurately presented, and the synchronization effect is also improved, which helps to convey voice information more clearly, so that it can be more widely used in virtual anchors, online conferences, games and other fields, providing a better interactive experience.

[0098] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A lip synchronization method for 3D digital human, characterized by: The following steps are included: S1. Audio data collection: Collect Chinese human voice audio, and manually proofread the collected audio to select audio files with standard pronunciation and clarity; S2, audio to text conversion: use the acoustic model and language model to process the audio file and the output of the acoustic model respectively; The decoding algorithm is used to combine the output of the language model and the acoustic model to obtain the text expression corresponding to the audio file; S3. Phoneme alignment: Use multimodal phoneme alignment tools to align audio files and their corresponding text expressions to generate phoneme time series; predict the corresponding lip state at each moment based on the phoneme time series to generate a lip animation time series; S4, mapping of phonemes to visemes: Based on the mapping relationship between the phoneme time series and the lip shape animation time series, the Chinese phoneme lip shapes are classified, and BlendShape is created for each type of lip shape according to the classification results. The BlendShape weight is set to represent the lip shapes of different phonemes, thus realizing the mapping of phonemes to lip shapes. S5. BlendShape weight calculation of mouth shape: based on the speech feature parameters of the audio, the BlendShape weight of the corresponding mouth shape is adjusted, the adjusted BlendShape weight is saved in the form of key frames and the number of key frames is optimized to a certain extent; S6. Unity modeling: Post-production artists and animation workers use Unity to model 3D tasks based on the weighted lip shape BlendShape and establish corresponding image models for each lip shape.

2. The lip synchronization method and system for 3D digital human according to claim 1, characterized in that: Step S2 specifically includes the following contents: S21, collecting audio signals in audio files and preprocessing them; S22, extracting key acoustic features from the preprocessed audio signal; S23, analyzing and modeling the extracted acoustic features using an acoustic model to predict speech units that may be contained in the audio; S24, using the language model to constrain and optimize the prediction results of the acoustic model based on the grammar, semantics and statistical laws of the language; S25. Based on the output of the decoding algorithm integrated acoustic model and language model, the predicted speech unit is converted into complete and accurate text content, and the text expression corresponding to the audio file is generated by searching for the optimal path or combination; the text expression adopts the pinyin form with numbers representing the tones.

3. The lip synchronization method and system for 3D digital human according to claim 1, characterized in that: Step S3 specifically includes the following contents: S31, analyzing and extracting features from the audio signal in the audio file to obtain key features of the audio; S32, parsing and encoding the text expression corresponding to the audio file, converting each pinyin character of the text expression and its corresponding tone number into a format that is convenient for calculation and comparison; S33, using a multimodal phoneme alignment tool to find the best correspondence between the key features of the audio and the text expression after the conversion format, and gradually find the optimal matching path through dynamic planning, so as to determine the exact position of each phoneme on the time axis and generate a phoneme time series; S34. Model the lip movements according to the characteristics and changing rules of the phonemes. Based on the generated phoneme time series, use the constructed model to predict the lip state corresponding to each moment, and generate a time series of lip animation.

4. The lip synchronization method and system for 3D digital human according to claim 1, characterized in that: Step S4 specifically includes the following contents: S41, based on the mapping relationship between the phoneme time series and the lip shape animation time series, classifying the lip shapes of the phonemes according to the pronunciation characteristics and visual similarity of various Chinese phonemes; S42, based on the mouth shape classification result of the phonemes, creating a mouth shape model of a specific mouth shape for each classification, that is, a specific type of BlendShape; S43. Set the weight of BlendShape to represent the mouth shapes corresponding to different phonemes.

5. The lip synchronization method and system for 3D digital human according to claim 4, characterized in that: Step S42 specifically includes: analyzing the common lip shape characteristics of the phonemes contained in each category, using 3D modeling software or animation production tools to create an initial basic lip shape model; adjusting and modifying the basic lip shape model based on the characteristics of each category to make it conform to the typical lip shape of the phonemes in the corresponding category; through multiple adjustments and optimizations, until a model that can accurately express the lip shape characteristics of the corresponding category phonemes, that is, a specific type of BlendShape, is created.

6. The lip synchronization method and system for 3D digital human according to claim 1, characterized in that: Step S5 specifically includes the following contents: S51, sampling the audio signal at a fixed sampling frequency according to the phoneme sequence, and calculating the speech feature parameters of the sampled audio; S52, according to the voice feature parameters, weighted adjustment is performed on the BlendShape weight of the corresponding mouth shape; S53, saving the weighted BlendShape weight after weighted adjustment as an animation curve in the form of key frames; S54. Optimize the number of key frames and reduce unnecessary key frames.

7. The lip synchronization method and system for 3D digital human according to claim 6, characterized in that: Step S52 specifically includes the following contents: S521, setting corresponding judgment thresholds and ranges for speech feature parameters related to the lip shape, including three ranges of high, medium and low; S522. Compare the sampled speech feature parameters with the judgment threshold of the corresponding lip shape classification to determine the range of the speech feature parameters. If it is in a low range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameter is slightly increased or remains unchanged; if it is in a medium range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameter is appropriately increased; if it is in a high range, the BlendShape weight of the lip shape classification corresponding to the speech feature parameter is greatly increased.

8. The lip synchronization method and system for 3D digital human according to claim 7, characterized in that: The speech feature parameters include volume, pitch, and speech speed.

9. The lip synchronization method and system for 3D digital human according to claim 7, characterized in that: When increasing the weight, the specific increase is calculated according to the pre-set rules or algorithms; in the process of adjusting the BlendShape weight, the comprehensive influence of multiple speech feature parameters should also be considered, and the adjustment results of multiple parameters are combined through weighted averaging and priority sorting to finally determine the specific adjustment value of the BlendShape weight for each mouth shape.

10. A lip synchronization system for 3D digital humans, characterized by: The system can implement the method described in any one of claims 1 to 9, and the system includes: Audio data collection module: collects Chinese human voice audio, and manually proofreads the collected audio to select audio files with standard pronunciation and clarity; Audio-to-text module: uses the acoustic model and language model to process the audio file and the output of the acoustic model respectively; The decoding algorithm is used to combine the output of the language model and the acoustic model to obtain the text expression corresponding to the audio file; Phoneme alignment module: Use multimodal phoneme alignment tools to align audio files and their corresponding text expressions to generate phoneme time series; predict the corresponding lip state at each moment based on the phoneme time series to generate a lip animation time series; Phoneme to viseme mapping module: Based on the mapping relationship between the phoneme time series and the lip shape animation time series, the Chinese phoneme lip shapes are classified, and BlendShape is created for each type of lip shape according to the classification results. The BlendShape weight is set to represent the lip shapes of different phonemes, thus realizing the mapping from phonemes to lip shapes. Lip shape BlendShape weight calculation module: Based on the voice feature parameters of the audio, the BlendShape weight of the corresponding lip shape is adjusted, the adjusted BlendShape weight is saved in the form of key frames and the number of key frames is optimized to a certain extent; Unity modeling module: Post-production artists and animators use Unity to model 3D tasks based on the weighted lip shape BlendShape, and establish corresponding image models for each lip shape.

Images