A method for enhancing the diversity of digital population based on cryptophone retrieval

Abstract

The present invention belongs to the technical field of digital population animation synthesis, and specifically relates to a method for enhancing the diversity of digital population based on hidden phoneme retrieval, which is used to enhance the authenticity and diversity of digital population animation. Specifically, in the traditional phoneme-viseme matching scheme, since the number of phoneme and viseme categories is limited by artificial definition, the phonemes extracted from the speech cannot fully capture the subtle dynamic changes in the pronunciation process, which in turn affects the natural fluency of the lip animation. In order to overcome this limitation, the present invention adopts deep learning technology to pre-process and extract features of the audio and video materials of digital people, and constructs a hidden phoneme-viseme library. In real-time interaction, the realism and coherence of the digital population animation are achieved through the retrieval and matching of audio hidden phonemes. The advantage of the present invention is that it can continuously optimize and iterate to improve the coherence and realism of the lip animation.

Description

Technical Field

[0001] The invention belongs to the technical field of digital population type animation synthesis, and in particular relates to a method for enhancing the diversity of digital population type based on cryptophone retrieval. Background Art

[0002] In the field of digital population animation synthesis, the existing mainstream technical solutions are usually based on the matching method of phonemes and visemes. This method first extracts a continuous phoneme sequence from the voice, such as: / ni3hɑo3 / (hello), (hello), and then select the corresponding visual element images according to the correspondence between the phonemes and the predefined visual elements, and finally splice these images into a smooth lip animation.

[0003] Phonemes are the smallest units of sound that can distinguish meaning in human language. They are usually segmented and classified by speech experts through manual annotation or using speech recognition tools. Visemene refers to the visual state of the pronunciation organ corresponding to a specific phoneme. The MPEG-4 standard proposes the concept of visemes and applies it to speech animation synthesis.

[0004] Commercial applications on the market usually adopt this solution to reduce costs and ensure real-time performance. First, the input voice signal is segmented at the phoneme level to identify each phoneme in the continuous speech. Then, the corresponding lip animation is determined based on the mapping relationship between phonemes and visemes. The advantage of this method is that it is relatively simple to implement and can quickly synchronize voice and lip animation. It is widely used on mobile phones with limited computing power.

[0005] However, existing technologies have some limitations. Since the mapping relationship between phonemes and visemes is based on static, discrete states, this method has difficulty capturing dynamic changes and nuances in the pronunciation process. In addition, the pronunciation characteristics and phoneme composition of different languages ​​vary greatly, resulting in existing phoneme-viseme matching schemes that may not be accurate or natural enough when processing certain languages.

[0006] In addition, the number and definition of phonemes are different for different languages. For example, English may need to distinguish more consonant visemes to reflect its rich consonant endings and consonant clusters; the initials and finals in Chinese are complex, and there are various methods to classify visemes, from classification based on initials and finals to grouping phonemes using clustering methods. Different classification methods reflect the complexity of Chinese pronunciation. For example, some studies divide Chinese into 28 basic static visemes, while Microsoft divides visemes into 22 to meet the needs of different languages. These classification methods provide the basis for the mapping of phonemes to visemes, and the generation of these phonemes requires linguistic rules, which are usually manually annotated by speech experts with relevant expertise or segmented and classified by advanced speech recognition tools. This is a very time-consuming process. At the same time, due to the limitations of manual definition, the limited number of discrete phonemes still faces challenges in pronunciation coherence and animation authenticity in practical applications.

[0007] In summary, although the existing phoneme-viseme matching schemes have been widely used in digital population animation synthesis, there is still room for improvement in capturing the dynamic changes of natural pronunciation and adapting to the characteristics of different languages. Summary of the invention

[0008] The purpose of the present invention is to provide a method for enhancing the diversity of digital population types based on cryptophone retrieval, which overcomes the limitations of the prior art and achieves more natural and coherent digital population animation synthesis by introducing cryptophone technology based on deep learning.

[0009] To achieve the above-mentioned purpose, the present invention adopts the following technical solution: a method for enhancing the diversity of digital population based on cryptophone retrieval, comprising the following steps:

[0010] a) Data preprocessing step, preprocessing the audio and video data in the existing digital human speaking video to provide the preprocessed audio and video data for step b);

[0011] b) a feature extraction step, using a deep audio feature encoder to extract features from the audio data preprocessed in step a) to generate a feature vector, and the feature vector generated in step b) is used as input to step c);

[0012] c) a step of generating hidden phonemes, using a clustering algorithm to cluster the audio feature vectors extracted in step b) to generate central feature vectors representing different hidden phonemes, and the hidden phonemes generated in step c) are used for constructing the hidden phoneme-viseme library in step d);

[0013] d) a step of constructing a cryptophone-viseme library, in which the cryptophones generated in step c) are associated with corresponding video picture frames to form a viseme library containing a variety of pronunciation mouth shapes, and the library constructed in step d) provides cryptophones and viseme maps for step e);

[0014] e) audio latent phoneme retrieval step: during the real-time interaction between the digital human and the user, the audio clip is converted into a feature vector through a deep audio feature encoder, and the distance or similarity with the latent phoneme set constructed in step d) is calculated to retrieve the most matching visual lip-sync image sequence, and the retrieval result of step e) is used in step f);

[0015] f) a viseme matching step, matching the corresponding viseme from the library constructed in step d) according to the hidden phoneme lip shape picture sequence retrieved in step e), and the viseme matched in step f) provides a lip shape sequence for step g);

[0016] g) a lip sequence synthesis step, combining the matched viseme images in step f) in a time sequence to generate a coherent digital mouth animation, the animation synthesized in step g) providing a basis for step h);

[0017] h) a dynamic matching mechanism step, in which, during real-time audio processing, the number and matching relationship of hidden phonemes and visemes are dynamically adjusted to optimize the expressiveness of lip animation, and the adjustment of step h) is based on the interactive experience feedback in step i);

[0018] i) Continuous optimization and iteration step: According to the pronunciation style of the digital human and the real interactive experience, the number of hidden phonemes and visemes is continuously adjusted to improve the adaptability and performance of the system. The feedback loop of step i) acts on steps c) to h).

[0019] Preferably, the data preprocessing step a) comprises the following sub-steps:

[0020] a1) an audio resampling step, resampling all audio data to 16,000 Hz, and the audio processed in step a1) provides input for step a2);

[0021] a2) Mel spectrogram calculation step, using a 25 millisecond window and a 10 millisecond step size to perform sliding segmentation on the audio data resampled in step a1), and calculate the logarithmic Mel spectrogram of 80 channels, and the Mel spectrogram generated in step a2) is used as the input of step a3);

[0022] a3) Video frame rate conversion step, converting the video to 25 frames per second and splitting the video into individual pictures to ensure that each frame corresponds to 40ms of audio data. The picture generated in step a3) and the Mel spectrum diagram in step a2) are used together as inputs of step b).

[0023] Preferably, the feature extraction step b) comprises the following sub-steps:

[0024] b1) extracting features from the Mel spectrogram data generated in step a2) using the encoder part of the Whisper model, which contains two convolutional layers, sinusoidal position encoding, and 4 standard Transformer encoder blocks;

[0025] b2) After processing in step b1), the Mel data of 30 seconds of audio is converted into a feature vector, and every 20ms of audio is converted into a 5*384-dimensional feature vector. The feature vector generated in step b2) is used as the input for generating latent phonemes in step c).

[0026] Preferably, the cryptophone generation step c) comprises the following sub-steps:

[0027] c1) performing data preprocessing on the feature vector generated in step b2), including normalizing or standardizing each feature;

[0028] c2) After completing the data preprocessing in step c1), a preset number of clusters K is selected, K centroids are initialized using the K-means clustering algorithm, and the feature vectors processed in step b2) are assigned to the nearest centroids to form K clusters, the centers of these clusters being the central feature vectors of the hidden phonemes;

[0029] c3) By repeatedly executing the operations of assigning data points to the nearest center and updating the centroid in step c2) until the change of the centroid is very small or a predetermined number of iterations is reached, the hidden phonemes finally generated in step c3) are used for the construction of the hidden phoneme-viseme library in step d).

[0030] Preferably, the cryptophone-viseme library construction step d) comprises the following sub-steps:

[0031] d1) associating the K latent phoneme center feature vectors generated in step c3) with corresponding video frames, where the video frames are extracted from the pre-processed digital human speaking video, and each video frame corresponds to a specific lip shape state;

[0032] d2) constructing a viseme library consisting of K pronunciation mouth shape pictures corresponding to the latent phonemes according to the correlation between the latent phoneme central feature vector and the video frame in step d1), wherein each viseme picture in the library represents the pronunciation state of a latent phoneme;

[0033] d3) Using the viseme library constructed in step d2), a hidden phoneme retrieval mechanism is implemented in step e), that is, the feature vector obtained during the real-time audio processing is compared with the hidden phonemes in the library, the best matching viseme is found and a coherent lip animation is synthesized.

[0034] Preferably, the audio hidden phoneme retrieval step e) comprises the following sub-steps:

[0035] e1) During the real-time interaction between the digital human and the user, the input audio clip is converted into a series of 5*384-dimensional feature vectors through the deep audio feature encoder (Whisper);

[0036] e2) for each feature vector obtained in step e1), calculating the Euclidean distance or cosine similarity with the latent phonemes in the latent phoneme set constructed in step d2);

[0037] e3) According to the distance or similarity calculated in step e2), the hidden phoneme that best matches the current audio clip is retrieved from the hidden phoneme set, and then the corresponding viseme lip-image sequence is determined.

[0038] Preferably, the viewme graph matching step f) comprises the following sub-steps:

[0039] f1) determining the corresponding viseme graphs according to the lip-image sequence of the hidden phoneme retrieved in step e3), wherein the viseme graphs are from the hidden phoneme-viseme graph library constructed in step d2);

[0040] f2) extracting the viseme graph determined in step f1) from the library in step d2) as a basic viseme graph set for synthesizing a coherent digital population-type animation in the next step g);

[0041] f3) The visual graphs extracted in step f2) are sorted and organized to form a coherent lip sequence.

[0042] Preferably, the lip sequence synthesis step g) comprises the following sub-steps:

[0043] g1) arranging the visual graphs organized in step f3) in chronological order, ensuring that each visual graph corresponds to the correct audio time point;

[0044] g2) using graphics synthesis technology to merge the sequence of visual images arranged in step g1) into a continuous animation stream, generating a realistic digital population animation, and providing basic materials for the final presentation in step h);

[0045] g3) Smoothing the animation stream generated in step g2) to enhance the realism of the digital human interaction in step h).

[0046] Preferably, the dynamic matching mechanism step h) includes the following sub-steps:

[0047] h1) In the real-time audio processing process, the hidden phoneme features extracted in step f) are used to dynamically retrieve the viseme graph matching the hidden phoneme features, and the number of hidden phonemes is adjusted according to the actual effect;

[0048] h2) combining the lip animation sequence synthesized in step g), observing its coherence and authenticity, dynamically adjusting the number of hidden phonemes and visemes according to actual application conditions, and continuously optimizing the lip animation;

[0049] h3) Collect feedback on the user interaction experience in step i), and fine-tune the number and matching strategy of hidden phonemes and visemes in step h2) based on the feedback information.

[0050] Preferably, the continuous optimization and iteration step i) comprises the following sub-steps:

[0051] i1) Collecting interactive data between digital humans and users, including but not limited to user feedback, system response time, and consistency evaluation of lip animation;

[0052] i2) Based on the data collected in step i1), the pronunciation style of the digital human and the user's interactive experience are analyzed, and the number of hidden phonemes generated in step c) and the hidden phoneme-viseme library constructed in step 4) are dynamically adjusted;

[0053] i3) applying the latent phonemes and viseme graphs adjusted in step i2) to the dynamic matching mechanism in step h) to optimize the expressiveness of the lip animation, and continuing to monitor user feedback and system performance, and looping steps i1) to i3) to continuously improve the adaptability and performance of the system.

[0054] Technical effects and advantages of the present invention: Compared with the prior art, the method for enhancing the diversity of digital population based on cryptophone retrieval proposed by the present invention has the following advantages:

[0055] The present invention preprocesses the audio and video material data of existing digital humans, extracts features, builds a latent phoneme-viseme library, and retrieves and matches the latent phonemes of the audio, and finally realizes the realistic and coherent lip animation synthesis of the digital humans. The present invention can continuously improve the coherence and authenticity of the lip animation through optimization and iteration, and realizes the real-time generation of lip animation consistent with the audio. It is not limited by the number and professionalism of traditional phonemes, and can dynamically adjust the number of latent phonemes and visemes according to the pronunciation style and real interactive experience of different digital humans, thereby improving the adaptability and performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a topological diagram of the digital population diversity enhancement method based on cryptophone retrieval of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] The present invention proposes a method for enhancing the diversity of digital human lip shape based on cryptophone retrieval, which aims to achieve realistic and coherent lip shape animation synthesis of digital human by preprocessing the audio and video data of existing digital human, extracting features, building cryptophone-viseme library, and cryptophone retrieval-matching of audio. The advantage of this solution is that it can continuously improve the coherence and authenticity of lip shape animation through optimization and iteration.

[0059] It mainly includes two key parts: first, construct a hidden phoneme-viseme library, and establish a library containing multiple viseme graphs by generating and mapping hidden phonemes; second, in the real-time interaction process, process the audio, and retrieve matching viseme graphs from the viseme library based on the extracted hidden phonemes, and synthesize a coherent lip sequence. The specific process of this solution is as follows Figure 1 shown.

[0060] Specifically, in this embodiment, a method for enhancing the diversity of digital population based on cryptophone retrieval includes the following steps:

[0061] a) Data preprocessing step, preprocessing the audio and video data in the existing digital human speaking video to provide the preprocessed audio and video data for step b);

[0062] The data preprocessing step further includes the following sub-steps:

[0063] a1) an audio resampling step, resampling all audio data to 16,000 Hz, and the audio processed in step a1) provides input for step a2);

[0064] a2) Mel spectrogram calculation step, using a 25 millisecond window and a 10 millisecond step size to perform sliding segmentation on the audio data resampled in step a1), and calculate the logarithmic Mel spectrogram of 80 channels, and the Mel spectrogram generated in step a2) is used as the input of step a3);

[0065] a3) Video frame rate conversion step, converting the video to 25 frames per second and splitting the video into individual pictures to ensure that each frame corresponds to 40ms of audio data. The picture generated in step a3) and the Mel spectrum diagram in step a2) are used together as inputs of step b).

[0066] Specifically, first, the audio and video data in the existing digital human speech video are preprocessed (all audio is resampled to 16,000Hz, and the 80-channel logarithmic Mel spectrogram is calculated. To calculate the Mel spectrogram, the audio data is first segmented using a 25-ms window and a 10-ms step. For example, 30 seconds of audio corresponds to 480,000 audio sample points, each 25-ms sliding window corresponds to 400 sample points, and a step of 10ms corresponds to 160 sample points. A total of 2,998 windows are obtained. Considering the boundary effect, zero padding will be performed (120 zero values ​​are filled in the front and back), and finally a total of 3,000 windows can be obtained, corresponding to a 3,000*80 Mel array). The audio part is resampled to 16kHz to improve processing efficiency while ensuring the quality of the voice signal. The video part only needs to be converted to 25 frames per second and split into single pictures to ensure that each frame of the picture can correspond to 40ms of audio data to achieve audio and video synchronization.

[0067] b) a feature extraction step, using a deep audio feature encoder to extract features from the audio data preprocessed in step a) to generate a feature vector, and the feature vector generated in step b) is used as input to step c);

[0068] The feature extraction step further includes the following sub-steps:

[0069] b1) extracting features from the Mel spectrogram data generated in step a2) using the encoder part of the Whisper model, which contains two convolutional layers, sinusoidal position encoding, and 4 standard Transformer encoder blocks;

[0070] b2) After processing in step b1), the Mel data of 30 seconds of audio is converted into a feature vector, and every 20ms of audio is converted into a 5*384-dimensional feature vector. The feature vector generated in step b2) is used as the input for generating latent phonemes in step c).

[0071] Specifically, the Whisper automatic speech recognition (ASR) model (a sequence-to-sequence model based on the Transformer architecture. The complete whisper model consists of two parts: an encoder and a decoder, which converts the input speech sequence into an output text sequence. The encoder is responsible for extracting speech features, and the decoder is responsible for generating prediction results. The neural network structure of the encoder consists of three parts: 1. Two convolutional layers, using convolutional filters with a convolution kernel size of 3 and a GELU activation function, with a convolution step of 2; 2. Adding sinusoidal position encoding to the output of the convolutional layer; 3. Finally, the speech features are output through a 4-layer standard Transformer encoder block.

[0072] The encoder of the Whisper model can process the [3000*80] Mel data extracted from the 30 seconds of audio mentioned above into a [1500*5*384] feature vector, where the 3000 dimensions are reduced to 1500 dimensions, meaning that every 20ms of audio is converted into a [5*384]-dimensional feature vector. These feature vectors capture the key information in the audio signal and lay the foundation for subsequent hidden phoneme retrieval.

[0073] c) a step of generating hidden phonemes, using a clustering algorithm to cluster the audio feature vectors extracted in step b) to generate central feature vectors representing different hidden phonemes, and the hidden phonemes generated in step c) are used for constructing the hidden phoneme-viseme library in step d);

[0074] The step of generating hidden phonemes further includes the following sub-steps:

[0075] c1) performing data preprocessing on the feature vector generated in step b2), including normalizing or standardizing each feature;

[0076] c2) After completing the data preprocessing in step c1), a preset number of clusters K is selected, K centroids are initialized using the K-means clustering algorithm, and the feature vectors processed in step b2) are assigned to the nearest centroids to form K clusters, the centers of these clusters being the central feature vectors of the hidden phonemes;

[0077] c3) By repeatedly executing the operations of assigning data points to the nearest center and updating the centroid in step c2) until the change of the centroid is very small or a predetermined number of iterations is reached, the hidden phonemes finally generated in step c3) are used for the construction of the hidden phoneme-viseme library in step d).

[0078] Specifically, based on the extracted audio feature vector, a clustering algorithm is used to generate latent phonemes. The clustering algorithm identifies the redundancy in the audio features and generates central feature vectors representing different latent phonemes by distinguishing categories with large differences, thereby reducing the redundancy of information. Various clustering algorithms provided by the Scikit-Learn toolkit can be directly used here, thereby reducing the algorithm knowledge threshold for users. Take the k-means algorithm in the toolkit as an example:

[0079] K-means is a widely used clustering algorithm for grouping data points into k clusters. The following is the process of clustering using the k-means algorithm, assuming that there are 1500 data points, each of which is a 5*384 matrix.

[0080] a. Data preprocessing

[0081] Use normalization or standardization to ensure that each feature has the same influence in the clustering process. The steps are as follows:

[0082] 1. Import the numpy library for mathematical operations on arrays and matrices.

[0083] 2. Generate a random normally distributed array data of shape (1500,5,384), which contains 1500 samples, each sample has 5 features, and each feature has 384 values.

[0084] 3. Start a loop, from 0 to N, processing each feature of data in turn. (N is the number of features)

[0085] 4. Normalize the i-th feature. The specific steps are:

[0086] - Calculate the mean of this feature.

[0087] - Calculate the standard deviation of this feature.

[0088] -Subtract the mean from each value of the feature and divide by the standard deviation to normalize the feature's values ​​to a distribution with a mean of 0 and a standard deviation of 1.

[0089] b. Select K value

[0090] Determine the number of clusters K to be formed, that is, the number of final latent phonemes we preset, which can be customized according to actual needs. We can assume 100 and then adjust it according to the actual effect.

[0091] c. Initialization Center

[0092] Randomly select K data points as the initial center (centroid). The steps are as follows:

[0093] 1. Import the KMeans clustering algorithm from the sklearn (scikit-learn) library for subsequent clustering operations.

[0094] 2. Set the number of clusters k=100, which means that the data is divided into 100 clusters.

[0095] 3. Initialize a KMeans clustering model, specify the number of clusters as 100, and set the random seed to ensure the reproducibility of the results.

[0096] 4. Reshape the data into a two-dimensional array of shape (1500, 1920) (each sample consists of 1920 eigenvalues), and then perform KMeans clustering on the reshaped data to divide the data into 100 clusters.

[0097] d. Assign data points to the nearest center

[0098] For each data point, calculate its distance to all centers (Euclidean distance or cosine similarity) and assign it to the nearest center.

[0099] e. Update Center

[0100] Recompute the center of each cluster, usually taking the mean of all data points belonging to that cluster.

[0101] f. Repeat steps d and e

[0102] The steps of assigning and updating the center are repeated until the change in the center is very small or a predetermined number of iterations is reached.

[0103] g. Clustering results

[0104] Ultimately, each data point is assigned to a cluster, and the clustering results can be used for further analysis or decision making.

[0105] d) a step of constructing a cryptophone-viseme library, in which the cryptophones generated in step c) are associated with corresponding video picture frames to form a viseme library containing a variety of pronunciation mouth shapes, and the library constructed in step d) provides cryptophones and viseme maps for step e);

[0106] The cryptophone-viseme library construction step further includes the following sub-steps:

[0107] d1) associating the K latent phoneme center feature vectors generated in step c3) with corresponding video frames, where the video frames are extracted from the pre-processed digital human speaking video, and each video frame corresponds to a specific lip shape state;

[0108] d2) constructing a viseme library consisting of K pronunciation mouth shape pictures corresponding to the latent phonemes according to the correlation between the latent phoneme central feature vector and the video frame in step d1), wherein each viseme picture in the library represents the pronunciation state of a latent phoneme;

[0109] d3) Using the viseme library constructed in step d2), a hidden phoneme retrieval mechanism is implemented in step e), that is, the feature vector obtained during the real-time audio processing is compared with the hidden phonemes in the library, the best matching viseme is found and a coherent lip animation is synthesized.

[0110] e) audio latent phoneme retrieval step: during the real-time interaction between the digital human and the user, the audio clip is converted into a feature vector through a deep audio feature encoder, and the distance or similarity with the latent phoneme set constructed in step d) is calculated to retrieve the most matching visual lip-sync image sequence, and the retrieval result of step e) is used in step f);

[0111] The step of retrieving audio phonemes further includes the following sub-steps:

[0112] e1) During the real-time interaction between the digital human and the user, the input audio clip is converted into a series of 5*384-dimensional feature vectors through the deep audio feature encoder;

[0113] e2) for each feature vector obtained in step e1), calculating the Euclidean distance or cosine similarity with the latent phonemes in the latent phoneme set constructed in step d2);

[0114] e3) According to the distance or similarity calculated in step e2), the hidden phoneme that best matches the current audio clip is retrieved from the hidden phoneme set, and then the corresponding viseme lip-image sequence is determined.

[0115] Specifically, during the real-time interaction between the digital human and the user, the audio clip is first converted into a feature vector through a deep audio feature encoder (same as feature extraction, 30 seconds of audio is extracted into [3000*80] Mel data, processed by the encoder into a [1500*5*384] feature vector, and a [5*384]-dimensional audio feature vector is obtained every 20ms) and the distance or similarity is calculated with the latent phonemes in the latent phoneme set to quickly retrieve the visual lip-sync image sequence that best matches the current audio clip.

[0116] Distance: Euclidean distance, for two vectors a and b, ;

[0117] Similarity: Cosine Similarity, cos(a,b)= ,in is the dot product of a and b, is the Euclidean norm of a vector (usually used to calculate the length (or modulus) of a vector, for example: for a vector v = [ , , ,..., ], the formula of its Euclidean norm is = ).

[0118] Lip-sync sequence synthesis

[0119] According to the retrieved hidden phoneme lip shape image sequence, a coherent digital mouth shape animation is combined in a time sequence. This process ensures that the lip shape movements of the digital person when pronouncing are highly consistent with the audio content, significantly improving the realism of the interactive experience.

[0120] f) a viseme matching step, matching the corresponding viseme from the library constructed in step d) according to the hidden phoneme lip shape picture sequence retrieved in step e), and the viseme matched in step f) provides a lip shape sequence for step g);

[0121] The view graph matching step further includes the following sub-steps:

[0122] f1) determining the corresponding viseme graphs according to the lip-image sequence of the hidden phoneme retrieved in step e3), wherein the viseme graphs are from the hidden phoneme-viseme graph library constructed in step d2);

[0123] f2) extracting the viseme graph determined in step f1) from the library in step d2) as a basic viseme graph set for synthesizing a coherent digital population-type animation in the next step g);

[0124] f3) The visual graphs extracted in step f2) are sorted and organized to form a coherent lip sequence.

[0125] g) a lip sequence synthesis step, combining the matched viseme images in step f) in a time sequence to generate a coherent digital mouth animation, the animation synthesized in step g) providing a basis for step h);

[0126] The lip sequence synthesis step further includes the following sub-steps:

[0127] g1) arranging the visual graphs organized in step f3) in chronological order, ensuring that each visual graph corresponds to the correct audio time point;

[0128] g2) using graphics synthesis technology to merge the sequence of visual images arranged in step g1) into a continuous animation stream, generating a realistic digital population animation, and providing basic materials for the final presentation in step h);

[0129] g3) Smoothing the animation stream generated in step g2) to enhance the realism of the digital human interaction in step h).

[0130] Through the hidden phoneme retrieval mechanism, the best matching viseme graph can be quickly found according to the real-time audio input, ensuring that the generated lip animation is highly consistent with the audio content, thereby enhancing the realism of the digital human interaction experience. The viseme graph library generated based on hidden phonemes provides a richer and more dynamic lip representation than traditional phonemes, breaking through the limitation of the number of traditional phonemes, without relying on professional linguistic knowledge and complex annotation work, making the lip animation more diverse and natural. Through continuous optimization and iteration, the system can adjust the number of hidden phonemes and visemes according to feedback in actual applications, which can not only better adapt to the pronunciation styles of different digital humans, but also improve the overall performance of the system, making the generation of lip animation smoother and more efficient. In the process of real-time audio processing, the hidden phoneme features are used to dynamically retrieve the matching viseme graph and synthesize a coherent lip animation. This mechanism improves the accuracy and naturalness of the lip shape of digital humans during interaction, making digital humans closer to the expression of real people.

[0131] In summary, this scheme effectively solves the problems of insufficient realism and poor continuity of lip animation in traditional phoneme-viseme matching schemes by constructing a hidden phoneme-viseme library and a dynamic matching mechanism, and significantly improves the application effect of digital human visual speech synthesis technology.

[0132] h) a dynamic matching mechanism step, in which, during real-time audio processing, the number and matching relationship of hidden phonemes and visemes are dynamically adjusted to optimize the expressiveness of lip animation, and the adjustment of step h) is based on the interactive experience feedback in step i);

[0133] The dynamic matching mechanism step further includes the following sub-steps:

[0134] h1) In the real-time audio processing process, the hidden phoneme features extracted in step f) are used to dynamically retrieve the viseme graph matching the hidden phoneme features, and the number of hidden phonemes is adjusted according to the actual effect;

[0135] h2) combining the lip animation sequence synthesized in step g), observing its coherence and authenticity, dynamically adjusting the number of hidden phonemes and visemes according to actual application conditions, and continuously optimizing the lip animation;

[0136] h3) Collect feedback on the user interaction experience in step i), and fine-tune the number and matching strategy of hidden phonemes and visemes in step h2) based on the feedback information.

[0137] Through the dynamic matching mechanism, the viseme graph that matches the extracted latent phoneme features can be quickly retrieved during real-time audio processing to generate a coherent and natural lip animation. This makes the digital human appear more realistic during the interaction with the user, improving the user experience. Dynamically adjusting the number of latent phonemes and visemes can ensure that the generated lip animation is both coherent and natural, avoiding the unnatural phenomena caused by static and discrete mapping in traditional phoneme-viseme matching schemes.

[0138] This solution allows the number of hidden phonemes and visemes, as well as the matching strategy, to be adjusted continuously according to actual application conditions, so that the system can continuously evolve, adapt to different application scenarios and user needs, and improve overall performance. By collecting feedback information from user interaction experience, the number of hidden phonemes and visemes and the matching strategy can be further fine-tuned to make the system closer to the actual needs of users and improve satisfaction.

[0139] In summary, this technical solution overcomes the limitations of traditional phoneme-viseme matching schemes by introducing deep learning-based latent phoneme technology, achieving a more natural and coherent digital population animation synthesis, while further enhancing the adaptability and performance of the system through dynamic matching mechanism and user feedback loop optimization.

[0140] i) Continuous optimization and iteration step: According to the pronunciation style of the digital human and the real interactive experience, the number of hidden phonemes and visemes is continuously adjusted to improve the adaptability and performance of the system. The feedback loop of step i) acts on steps c) to h).

[0141] The continuous optimization and iteration step further includes the following sub-steps:

[0142] i1) Collecting interactive data between digital humans and users, including but not limited to user feedback, system response time, and consistency evaluation of lip animation;

[0143] i2) Based on the data collected in step i1), the pronunciation style of the digital human and the user's interactive experience are analyzed, and the number of hidden phonemes generated in step c) and the hidden phoneme-viseme library constructed in step 4) are dynamically adjusted;

[0144] i3) applying the latent phonemes and viseme graphs adjusted in step i2) to the dynamic matching mechanism in step h) to optimize the expressiveness of the lip animation, and continuing to monitor user feedback and system performance, and looping steps i1) to i3) to continuously improve the adaptability and performance of the system.

[0145] By continuously collecting data on the interaction between users and digital humans and analyzing it, we can more accurately understand the needs and preferences of users. This adjustment based on user feedback helps improve the pronunciation style and lip animation of digital humans to better meet user expectations, thereby improving the user experience.

[0146] Since the solution includes a complete feedback loop from data collection to analysis to application adjustment, it can effectively respond to changes in different scenarios, such as changes in user preferences or new technological advances. This ensures that the system not only performs well during initial deployment, but also continues to optimize over time to remain competitive.

[0147] By monitoring indicators such as system response time and lip animation consistency, potential problems such as high latency or poor lip sync can be discovered and resolved in a timely manner. This not only improves the operating efficiency of the system, but also ensures a high-quality service level.

[0148] In summary, the present invention preprocesses the audio and video data of existing digital humans, extracts features, constructs a latent phoneme-viseme library, and retrieves and matches the latent phonemes of the audio, and finally realizes the synthesis of realistic and coherent lip animation of the digital humans. The coherence and authenticity of the lip animation can be continuously improved through optimization and iteration, and the real-time generation of lip animation consistent with the audio can be realized. It is not limited by the number and professionalism of traditional phonemes, and the number of latent phonemes and visemes can be dynamically adjusted according to the pronunciation style and real interactive experience of different digital humans, thereby improving the adaptability and performance of the system.

[0149] Definitions of Abbreviations and Key Terms

[0150] Visual speech synthesis: also known as speech animation synthesis, refers to synthesizing a facial image sequence corresponding to a given text or speech. It has wide applications in digital humans, human-computer interaction, film and television entertainment and other fields.

[0151] Phoneme: The smallest unit of sound in human language that can distinguish meaning.

[0152] Viseme: refers to the state of the visual pronunciation organs such as the mouth, tongue, and jaw corresponding to a certain phoneme.

[0153] Deep Audio Feature Encoder: A deep neural network model trained to extract audio features based on deep learning technology, such as Whisper, automatic speech recognition (ASR) system.

[0154] Implictphoneme: A new concept proposed in this embodiment is a high-level audio representation extracted based on a deep audio feature encoder, which can replace traditional phonemes and provide richer information for lip animation synthesis.

[0155] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for enhancing the diversity of digital population based on cryptophone retrieval, characterized in that: The following steps are involved: a) Data preprocessing step, preprocessing the audio and video data in the existing digital human speaking video to provide the preprocessed audio and video data for step b); b) a feature extraction step, using a deep audio feature encoder to extract features from the audio data preprocessed in step a) to generate a feature vector, and the generated feature vector is used as input to step c); c) a step of generating hidden phonemes, using a clustering algorithm to cluster the audio feature vectors extracted in step b) to generate central feature vectors representing different hidden phonemes, and the generated hidden phonemes are used for constructing a hidden phoneme-viseme library in step d); d) a step of constructing a cryptophone-viseme library, in which the cryptophones generated in step c) are associated with corresponding video picture frames to form a viseme library containing a variety of pronunciation mouth shapes, and the constructed library provides cryptophones and viseme maps for step e); e) audio latent phoneme retrieval step: during the real-time interaction between the digital human and the user, the audio clip is converted into a feature vector through a deep audio feature encoder, and the distance or similarity with the latent phoneme set constructed in step d) is calculated to retrieve the most matching viseme lip-sync image sequence, and the retrieval result is used in step f); f) a viseme matching step, matching the corresponding viseme from the library constructed in step d) according to the hidden phoneme lip shape picture sequence retrieved in step e), and the matched viseme provides a lip shape sequence for step g); g) a lip sequence synthesis step, combining the matched viseme images in step f) in a time sequence to generate a coherent digital mouth animation, the synthesized animation providing a basis for step h); h) a dynamic matching mechanism step, in which, during real-time audio processing, the number and matching relationship of hidden phonemes and visemes in step d) are dynamically adjusted to optimize the expressiveness of the lip animation, and the adjustment of step h) is based on the interactive experience feedback in step i); i) Continuous optimization and iteration steps, according to the pronunciation style of the digital human and the real interactive experience, the number of hidden phonemes and visemes is continuously adjusted to improve the adaptability and performance of the system. The feedback loop of step i) acts on steps c) to h); The data preprocessing step a) comprises the following sub-steps: a1) audio resampling step, resampling all audio data to 16,000 Hz, and the processed audio provides input for step a2); a2) Mel spectrogram calculation step, using a 25 millisecond window and a 10 millisecond step size to perform sliding segmentation on the audio data resampled in step a1), and calculate the logarithmic Mel spectrogram of 80 channels, and the generated Mel spectrogram is used as the input of step a3); a3) Video frame rate conversion step, converting the video to 25 frames per second, and splitting the video into individual pictures to ensure that each frame of the picture can correspond to 40ms of audio data, and the generated picture and the Mel spectrum graph of step a2) are used as the input of step b); The feature extraction step b) comprises the following sub-steps: b1) extracting features from the Mel spectrogram data generated in step a2) using the encoder part of the Whisper model, which contains two convolutional layers, sinusoidal position encoding, and 4 standard Transformer encoder blocks; b2) After processing in step b1), the Mel data of 30 seconds of audio is converted into a feature vector, and each 20ms of audio is converted into a 5*384-dimensional feature vector. The generated feature vector is used as the input for generating latent phonemes in step c); The step c) of generating hidden phonemes comprises the following sub-steps: c1) performing data preprocessing on the feature vector generated in step b2), including normalizing or standardizing each feature; c2) After completing the data preprocessing in step c1), a preset number of clusters K is selected, K centroids are initialized using the K-means clustering algorithm, and the feature vectors processed in step b2) are assigned to the nearest centroids to form K clusters, the centers of these clusters being the central feature vectors of the hidden phonemes; c3) by repeatedly performing the operations of assigning data points to the nearest center and updating the centroid in step c2) until the change of the centroid is very small or a predetermined number of iterations is reached, the finally generated hidden phoneme is used for the construction of the hidden phoneme-viseme library in step d); The cryptophone-viseme library construction step d) comprises the following sub-steps: d1) associating the K latent phoneme center feature vectors generated in step c3) with corresponding video frames, where the video frames are extracted from the pre-processed digital human speaking video, and each video frame corresponds to a specific lip shape state; d2) constructing a viseme library consisting of K pronunciation mouth shape pictures corresponding to the latent phonemes according to the correlation between the latent phoneme central feature vector and the video frame in step d1), wherein each viseme picture in the library represents the pronunciation state of a latent phoneme; d3) using the viseme library constructed in step d2), implementing a hidden phoneme retrieval mechanism in step e), that is, comparing the feature vector obtained in the real-time audio processing process with the hidden phonemes in the library, finding the most matching viseme and synthesizing a coherent lip animation; The audio hidden phoneme retrieval step e) comprises the following sub-steps: e1) During the real-time interaction between the digital human and the user, the input audio clip is converted into a series of 5*384-dimensional feature vectors through the deep audio feature encoder; e2) for each feature vector obtained in step e1), calculating the Euclidean distance or cosine similarity with the latent phonemes in the latent phoneme set constructed in step d2); e3) according to the distance or similarity calculated in step e2), the most matching hidden phoneme with the current audio clip is retrieved from the hidden phoneme set, and then the corresponding viseme lip-image sequence is determined; The viewme graph matching step f) comprises the following sub-steps: f1) determining the corresponding viseme graphs according to the lip-image sequence of the hidden phoneme retrieved in step e3), wherein the viseme graphs are from the hidden phoneme-viseme graph library constructed in step d2); f2) extracting the viseme graph determined in step f1) from the library in step d2) as a basic viseme graph set for synthesizing a coherent digital population-type animation in the next step g); f3) The visual graphs extracted in step f2) are sorted and organized to form a coherent lip sequence.

2. According to the method for enhancing the diversity of digital population based on cryptophone retrieval according to claim 1, it is characterized in that: The lip sequence synthesis step g) comprises the following sub-steps: g1) arranging the visual graphs organized in step f3) in chronological order, ensuring that each visual graph corresponds to the correct audio time point; g2) using graphics synthesis technology to merge the sequence of visual images arranged in step g1) into a continuous animation stream, generating a realistic digital population animation, and providing basic materials for the final presentation in step h); g3) Smoothing the animation stream generated in step g2) to enhance the realism of the digital human interaction in step h).

3. A method for enhancing the diversity of digital population based on cryptophone retrieval according to claim 2, characterized in that: The dynamic matching mechanism step h) includes the following sub-steps: h1) In the real-time audio processing process, the hidden phoneme features extracted in step f) are used to dynamically retrieve the viseme graph matching the hidden phoneme features, and the number of hidden phonemes is adjusted according to the actual effect; h2) combining the lip animation sequence synthesized in step g), observing its coherence and authenticity, dynamically adjusting the number of hidden phonemes and visemes according to actual application conditions, and continuously optimizing the lip animation; h3) Collect feedback on the user interaction experience in step i), and fine-tune the number and matching strategy of hidden phonemes and visemes in step h2) based on the feedback information.

4. A method for enhancing the diversity of digital population based on cryptophone retrieval according to claim 3, characterized in that: The continuous optimization and iteration step i) comprises the following sub-steps: i1) Collecting interactive data between digital humans and users, including but not limited to user feedback, system response time, and consistency evaluation of lip animation; i2) Based on the data collected in step i1), the pronunciation style of the digital human and the user's interactive experience are analyzed, and the number of hidden phonemes generated in step c) and the hidden phoneme-viseme library constructed in step 4) are dynamically adjusted; i3) applying the adjusted latent phonemes and viseme graphs in step i2) to the dynamic matching mechanism in step h) to optimize the expressiveness of the lip animation, and continuing to monitor user feedback and system performance, and looping through steps i1) to i3).

Images