A method and system for real-time conversion of streaming text to digital humans

Abstract

This application provides a real-time conversion method and system for streaming text to digital human, relating to the field of digital human technology. This application uses a TTS engine to convert real-time received streaming text data into first audio data, then performs format conversion on the first audio data to obtain second audio data. Subsequently, the second audio data is segmented according to a preset segment size to generate multiple audio segments. Audio feature vectors are extracted from each audio segment, and these vectors are accumulated in chronological order to form a feature sequence. Based on this sequence, a corresponding digital human image frame is generated. Finally, the digital human image frame and the corresponding audio segment are output synchronously to form a digital human video. This method enables low-latency real-time conversion from streaming text to digital human video, has a complete end-to-end process to simplify development, and supports multiple TTS engines to adapt to different scenarios, meeting real-time interaction requirements.

Description

Technical Field

[0001] This application relates to the field of digital human technology, and in particular to a real-time conversion method and system for converting streaming text to digital humans. Background Technology

[0002] With the popularization of digital human technology, real-time conversion of streaming text into digital humans has become a core requirement. This technology can be widely used in scenarios such as virtual anchors, intelligent customer service, and online interaction, and its application prospects are very broad.

[0003] There is an existing streaming text-to-digital human conversion technology that first receives a complete text segment of a certain length, converts it into a complete audio file through a TTS engine, then extracts features from the audio as a whole and generates digital human image frames, and finally integrates them into a video output.

[0004] This method requires waiting for the text and audio data to be complete before initiating subsequent processes, resulting in a long interval between text input and video output, making it difficult to meet the needs of highly time-sensitive interactive scenarios. Therefore, existing technologies suffer from insufficient real-time response capabilities in streaming text-to-digital human conversion. Summary of the Invention

[0005] The purpose of this application is to provide a real-time conversion method and system for streaming text to digital human, so as to solve the problem of insufficient real-time response capability of streaming text to digital human conversion in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a real-time conversion method for streaming text to a digital human, comprising: The real-time received streaming text data is converted into first audio data through the TTS engine; The first audio data is converted to generate the second audio data. The second audio data is divided into segments according to a preset segment size to generate multiple audio segments; For each audio segment, audio features are extracted to generate an audio feature vector; Based on the feature sequence formed by accumulating the audio feature vectors in chronological order, a digital human image frame corresponding to the feature sequence is generated; The digital human image frames and corresponding audio segments are output synchronously to form a digital human video.

[0007] Optionally, generating a digital human image frame corresponding to the feature sequence formed by accumulating the audio feature vectors in chronological order includes: A feature vector buffer queue is set up, which is used to cache the audio feature vectors sequentially according to the generation order; Each audio feature vector is added sequentially to the feature vector buffer queue; Monitor the number of audio feature vectors accumulated in the feature vector buffer queue; When the number of audio feature vectors accumulated in the feature vector buffer queue reaches a second preset threshold, multiple consecutive audio feature vectors that have reached the second preset threshold are extracted from the buffer queue as a feature sequence, and the feature sequence is input into the digital human driving model to generate a digital human image frame. The second preset threshold is 8 frames of audio feature vectors. After generating the digital human image frame, the feature vector buffer queue is updated to continue caching subsequently generated audio feature vectors.

[0008] Optionally, the step of inputting the feature sequence into the digital human driving model to generate a digital human image frame includes: The feature sequence is input into the digital human driving model; Within the digital human driving model, frame-by-frame feature analysis is performed on the input feature sequence to extract temporal variation parameters related to the digital human's facial expressions, lip movements, and head posture. Based on the time-series change parameters, in the digital human driving model, the preset digital human 3D model or digital human 2D image template is driven to perform corresponding deformations and movements; The three-dimensional model of the digital human after deformation and motion is rendered, or the two-dimensional image template of the digital human is synthesized to finally generate a digital human image frame with corresponding visual actions.

[0009] Optionally, the step of extracting audio features from each audio segment to generate an audio feature vector includes: A sample collection window is set up, which is used to collect the samples contained in the audio segment in the order of reception; The samples contained in each of the audio segments are added sequentially to the sample collection window; The total number of samples accumulated in the sample collection window is determined in real time; When the total number of samples accumulated in the sample collection window reaches a first preset threshold, feature calculation processing is performed on all samples accumulated in the current window to generate an audio feature vector. The first preset threshold is 11040 samples. After generating the audio feature vector, the sample collection window is updated to continue collecting samples from subsequent audio segments.

[0010] Optionally, the audio segment size is 800 samples, corresponding to an audio duration of 50 milliseconds at a sampling rate of 16kHz.

[0011] Optionally, the second audio data is in 16-bit PCM format, with a sampling frequency of 16kHz and is mono.

[0012] Optionally, the TTS engine is edge-tts.

[0013] Secondly, this application provides a real-time conversion system for streaming text to digital human, comprising: The text conversion module is used to convert real-time received streaming text data into first audio data through the TTS engine; The format conversion module is used to perform format conversion processing on the first audio data to generate the second audio data; The audio segmentation module is used to segment the second audio data according to a preset segmentation size to generate multiple audio segments; The feature extraction module is used to extract audio features from each audio segment and generate an audio feature vector. The image generation module is used to generate a digital human image frame corresponding to the feature sequence formed by accumulating the audio feature vector in chronological order. The synchronous output module is used to synchronously output the digital human image frames and the corresponding audio segments to form a digital human video.

[0014] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the real-time conversion method from streaming text to digital human as described in the first aspect above.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the real-time conversion method from streaming text to digital human as described in the first aspect above.

[0016] The real-time text-to-digital human conversion method provided in this application converts streaming text into first audio data using a TTS engine, achieving real-time text-to-audio conversion and providing basic audio materials for subsequent processes. It then converts the first audio data to generate second audio data, unifying the audio data format to adapt to subsequent segmentation and feature extraction requirements. The second audio data is then segmented into audio segments according to a preset size, achieving continuous audio splitting and meeting the unitization requirements of real-time processing. Audio feature vectors are extracted from the audio segments to uncover key audio information, providing core data support for digital human image frame generation. Digital human image frames are generated based on temporal feature sequences, achieving a mapping between audio features and digital human visual actions. Finally, digital human image frames and audio segments are output synchronously, achieving temporal matching of audio and video to form a complete and usable digital human video.

[0017] Furthermore, when generating digital human image frames, a feature vector buffer queue is first set up to cache audio feature vectors in the order of generation. Each audio feature vector is added to the queue sequentially, and the number of vectors accumulated in the queue is monitored in real time. When the number reaches a second preset threshold of 8 frames, the consecutive audio feature vectors of that number are extracted from the queue as a feature sequence and input into the digital human driving model to generate a digital human image frame. After the image frame is generated, the feature vector buffer queue is updated to continue caching subsequently generated audio feature vectors. Through the caching and threshold control of the feature vector buffer queue, the temporal continuity of the feature sequence input to the digital human driving model is ensured, ensuring that the generated digital human image frame has smooth and natural movements. At the same time, parallel processing of feature vector caching and image frame generation is realized, maintaining the continuity of streaming data processing and avoiding interruption of subsequent feature vector caching. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a real-time conversion method from streaming text to digital human provided in an embodiment of this application; Figure 2 A flowchart illustrating another real-time method for converting streaming text to a digital human, provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a real-time conversion system from streaming text to digital human provided in an embodiment of this application. Detailed Implementation

[0020] In real-time text-to-digital human conversion applications, existing technologies require first receiving a complete text segment of a certain length, converting it into a complete audio file via a TTS engine, then extracting features from the audio and generating digital human image frames, finally integrating them into a video output. This processing mode, which requires waiting for the text and audio data to be fully ready before starting subsequent processes, results in a long interval between text input and video output, making it difficult to meet the high-time-efficiency interaction requirements of scenarios such as virtual anchors and intelligent customer service.

[0021] To address the aforementioned issues, this application proposes a real-time conversion method for streaming text to digital humans. The core of this method lies in the segmented real-time processing of streaming data throughout the entire process. This method converts the received streaming text into audio data segment by segment, unifies the format, and then splits it into standardized audio segments. Feature vectors are extracted from each segment, and digital human image frames are generated based on temporal feature sequences. Finally, the image frames and corresponding audio segments are output synchronously to form a video. This method eliminates the need to wait for complete text and audio data, achieving low-latency conversion from text to digital human video through streaming segmentation processing. It fundamentally solves the problem of insufficient real-time response capability in existing technologies and meets the application requirements of various real-time interactive scenarios.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The core of this application is to provide a real-time method for converting streaming text to a digital human, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes: S101. The real-time received streaming text data is converted into first audio data through the TTS engine.

[0024] The TTS engine is edge-tts.

[0025] Streaming text data is the core input data processed by this method. Specifically, it includes text data transmitted in real time and continuously, in units of characters, sentences, or paragraphs. Unlike traditional one-time input of complete text, the transmission of this type of data does not require waiting for all text content to be prepared. Instead, it is transmitted as it is generated, providing a continuous source of input for subsequent audio conversion and digital human generation processes. This is the fundamental prerequisite for ensuring the real-time performance of the entire method.

[0026] In one specific implementation, the core objective of this step is to complete the real-time conversion from text to audio, providing the audio foundation for the subsequent generation of digital human image frames. The specific process is as follows: First, it receives streaming text data transmitted from external sources in real time, such as real-time bullet screen text in virtual anchor scenarios and user input text in intelligent customer service scenarios. Then, it calls the preset TTS engine to process these continuously transmitted texts in real time.

[0027] The TTS engine converts the text content into corresponding audio signals segment by segment according to the order of text transmission. This audio data is the first audio data generated in this step. The entire conversion process is synchronized with the text transmission process and does not wait for subsequent text input. It should be noted that the TTS engine selected in this step can preferably be the edge-tts service, which is lightweight, has a fast response speed, and supports streaming processing, making it well-suited to the real-time requirements of this method. At the same time, other text-to-speech services with similar functions can also be selected according to the deployment environment and performance requirements of the actual application scenario to ensure the stability and flexibility of the conversion function.

[0028] S102. Perform format conversion processing on the first audio data to generate the second audio data.

[0029] The second audio data is in 16-bit PCM format with a sampling frequency of 16kHz and is mono. The 16-bit PCM format is a digital audio format that directly samples and encodes the audio signal, offering advantages such as simple data structure and fast processing speed. The 16kHz sampling frequency and mono attribute are parameters chosen to balance audio clarity and data processing efficiency. Together, these three elements constitute a standard audio format suitable for subsequent audio segmentation and feature extraction processes.

[0030] In one specific implementation, the core objective of this step is to standardize the audio data format, eliminate differences in output formats between different TTS engines, and provide standardized input for subsequent processes. The specific process is as follows: After receiving the first audio data generated in step S101, the format conversion program is started. According to preset rules, the encoding format, sampling frequency, and number of channels of the first audio data are converted, ultimately generating second audio data that meets the requirements of 16-bit PCM format, 16kHz sampling frequency, and mono. The entire conversion process is synchronized with the audio data reception process, without any additional waiting delay. This ensures the real-time performance of streaming processing and lays the foundation for the efficient execution of subsequent steps.

[0031] S103. Divide the second audio data into segments according to a preset segment size to generate multiple audio segments.

[0032] The audio segmentation is a small audio unit obtained by splitting the continuous second audio data into segments of a fixed size. This unit is used to adapt to the subsequent segment-by-segment audio feature extraction process and ensure the continuity of streaming processing. The preset segment size refers to the fixed data volume standard set when splitting the audio. In this step, the segment size of 800 samples corresponds to an audio duration of 50 milliseconds at a sampling rate of 16kHz. This ensures that each segment contains enough audio information to support feature extraction and also ensures the real-time performance of the overall processing by splitting the segments into segments of a short duration, avoiding delays caused by excessively large segments.

[0033] In one specific implementation, the core objective of this step is to break down continuous, standardized audio data into small, parallel-processable units, providing a suitable input format for subsequent real-time feature extraction. The specific process is as follows: First, the system receives the continuous second audio data generated in step S102. Following a preset standard of 800 samples / segment, the audio data is sequentially split, with each accumulated 800 samples forming an independent audio segment. Multiple ordered audio segments are continuously generated and sequentially passed to subsequent steps. The entire segmentation process proceeds synchronously with the generation of the second audio data, eliminating the need to wait for complete audio data and ensuring uninterrupted real-time streaming processing. Simultaneously, the fixed segment size ensures the standardization of subsequent feature extraction processes.

[0034] S104. Extract audio features from each audio segment to generate an audio feature vector.

[0035] S104 specifically includes: S1041. Set a sample collection window, which is used to collect the samples contained in the audio segment in the order of reception.

[0036] The sample collection window is a temporary data buffer unit adapted to streaming processing. It is used to collect samples contained in audio segments in the order they are received, so as to avoid incomplete feature extraction due to insufficient sample size in a single segment.

[0037] S1042. Add the samples contained in each audio segment to the sample collection window in sequence.

[0038] S1043. Determine the total number of samples accumulated in the sample collection window in real time.

[0039] S1044. When the total number of samples accumulated in the sample collection window reaches a first preset threshold, feature calculation processing is performed on all samples accumulated in the current window to generate an audio feature vector.

[0040] The first preset threshold is 11,040 samples. This threshold is set to balance the integrity of audio features with the real-time performance of streaming processing, ensuring that the extracted features fully reflect a valid audio segment while avoiding delays caused by waiting for too many samples. Audio feature vectors are data carriers extracted from audio data that represent key sound information. In the real-time conversion scenario from streaming text to digital humans, they are used to establish the correlation between audio and the visual actions of the digital human, providing core data support for the subsequent generation of digital human image frames.

[0041] S1045. After generating the audio feature vector, update the sample collection window to continue collecting samples from subsequent audio segments.

[0042] In one specific implementation, this step achieves continuous extraction of streaming audio features through a closed-loop process of window setting, sample accumulation, threshold judgment, feature calculation, and window updating. The overall process is as follows: first, set a sample collection window adapted to streaming processing; then, store the continuously generated audio segment samples into the window in sequence; monitor the total number of samples in the window in real time; when the sample volume reaches the target, trigger feature calculation to generate audio feature vectors; and finally, update the window to continuously receive subsequent samples.

[0043] As an example, let's take the real-time processing of audio segments corresponding to the viewer's live chat text as an example for detailed explanation: First, a sample collection window with a capacity of 11,040 samples is set up in step S1041. The core function of the window is to collect audio segment samples in sequence and provide a sufficient number of samples for feature extraction.

[0044] Secondly, through step S1042, the samples contained in each audio segment generated in step S103 are added to the sample collection window in the order of reception, for example, 800 samples of the first segment, 800 samples of the second segment, and so on, and continuously stored in the window.

[0045] Next, in step S1043, the total number of samples in the window is counted in real time. The count is updated and it is checked whether 11,040 samples have been reached each time a slice is added. When the 14th slice is added, the first 13 slices have a total of 10,400 samples, the 14th slice has 800 samples, and the total number of samples is 11,200, which slightly exceeds the threshold of 11,040 samples. Therefore, feature calculation processing is triggered in step S1044.

[0046] This embodiment uses the Mel-frequency cepstral coefficients (MFCC) algorithm for audio feature extraction. This algorithm can effectively capture key information such as timbre and pitch of audio, and is suitable for the lip-shape and expression-driven requirements of digital humans. The specific implementation process is as follows: The first step is to pre-emphasize the 11,040 samples within the window to enhance the high-frequency signal, using the following formula: (1) In the formula, These are the original sample values. The values ​​are the sample values ​​after pre-emphasis, and 0.97 is the pre-emphasis coefficient. The pre-emphasis coefficient of 0.97 is a commonly used empirical value in the industry, used to balance the high-frequency enhancement effect and signal distortion.

[0047] For example, if the original sample x(100) = 0.5 and the previous sample x(99) = 0.48, substituting these values ​​into equation (1) will result in the pre-weighted value. .

[0048] The second step involves dividing the 11,040 pre-emphasized samples into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. At a sampling rate of 16kHz, each frame contains 16,000 × 0.025 = 400 samples, and the number of shifted samples is 16,000 × 0.01 = 160. The total number of frames is calculated as follows: (2) In the formula, This is the floor function.

[0049] The third step is to perform a 512-point Fast Fourier Transform (FFT) on each frame sample to convert the time-domain signal into a frequency-domain signal.

[0050] The fourth step involves filtering the frequency domain signal using 26 Mel filter banks to extract Mel frequency features and obtain the Mel spectrum feature vector for each frame.

[0051] The fifth step is to perform Discrete Cosine Transform (DCT) on the Mel spectrum feature vector, and take the first 13 coefficients as MFCC features to finally generate an audio feature vector with a dimension of 13×67, 13 MFCC coefficients, and 67 frames.

[0052] After the feature vector is generated, the sample collection window is updated in step S1045, retaining the last 480 samples (11040) within the window. 160×(67 1) = 11040 (10560=480) overlaps with subsequent new samples to ensure temporal continuity. Remove the remaining processed samples and continue to receive samples from the 15th and subsequent audio segments, repeating the above process.

[0053] The above example is only one example of this application. In practical applications, the feature extraction algorithm can be adjusted according to the needs, such as using Mel spectrograms, fundamental frequency features, etc., and the first preset threshold can also be adjusted. This application does not limit this.

[0054] In another specific implementation, a combination of Mel spectrograms and fundamental frequency extraction algorithms can be used for audio feature extraction. This combination can retain richer audio frequency domain details and pitch variation information, and is more suitable for the precise driving of digital human facial expressions and lip movements. It is especially suitable for scenarios that require high precision in digital human movements, such as real-time virtual customer service and virtual anchor interaction scenarios. The specific implementation process is as follows: First, pre-emphasis processing is performed on the audio data that reaches the first preset threshold within the sample collection window. The pre-emphasis formula consistent with the aforementioned equation (1) is used to enhance the high-frequency signal and avoid feature distortion caused by high-frequency attenuation. Then, the pre-emphasized samples are framed according to a frame length of 20 milliseconds and a frame shift of 10 milliseconds. At this time, each frame has 320 samples and a frame shift of 160 samples at a sampling rate of 16kHz. Similarly, the total number of frames is calculated as follows: frame.

[0055] Next, a 512-point Fast Fourier Transform is performed on each frame sample to convert the time-domain signal into a frequency-domain signal. Then, the frequency-domain signal is filtered by a 32-Mel filter bank to obtain the Mel spectrum features of each frame, with a dimension of 32×1. At the same time, the autocorrelation method is used to extract the fundamental frequency features of each frame. By calculating the autocorrelation function of the sample sequence, the delay step corresponding to the peak of the autocorrelation function is found, and the fundamental frequency value is converted to form a 1-dimensional fundamental frequency feature of each frame.

[0056] Finally, the 32-dimensional Mel-spectrum features of each frame are concatenated with the 1-dimensional fundamental frequency features to obtain a 33-dimensional fused feature per frame. Then, the fused features of 68 frames are temporally normalized to generate an audio feature vector with a dimension of 33×68. The window update mechanism is consistent with the above, preserving the overlap between the last 480 samples within the window and subsequent new samples to ensure the temporal continuity of the features.

[0057] This implementation enhances the representational capability of features through multi-dimensional feature fusion, enabling more accurate matching of subtle changes in digital human facial movements. It is suitable for streaming conversion scenarios with high requirements for interactive finesse. The above example is merely one illustration of this application. In practical applications, parameters such as the number of filters, frame length, and frame shift can be adjusted according to the needs of the digital human driver; this application does not impose any limitations on this.

[0058] This application realizes real-time and continuous extraction of streaming audio features, which not only ensures the integrity and accuracy of audio features, but also maintains the temporal continuity of features through a sliding window mechanism, avoiding the feature discontinuity problem in streaming processing, and providing high-quality data support for the subsequent generation of smooth digital human image frames based on feature sequences.

[0059] S105. Generate a digital human image frame corresponding to the feature sequence formed by accumulating the audio feature vectors in chronological order.

[0060] S105 specifically includes: S1051. Set up a feature vector buffer queue, which is used to cache the audio feature vectors sequentially according to the generation order.

[0061] The feature sequence refers to a set of multiple audio feature vectors arranged in chronological order. In this step, it specifically refers to eight consecutive audio feature vectors.

[0062] S1052. Each of the audio feature vectors is added sequentially to the feature vector buffer queue.

[0063] The feature vector buffer queue has a capacity that adapts to the requirements of the second preset threshold. It can temporarily store at least 8 audio feature vectors and supports ordered enqueue and dequeue operations for vectors. Its core design is to ensure the temporal continuity of audio feature vectors and provide a stable data source for the extraction of subsequent feature sequences.

[0064] S1053. Monitor the number of audio feature vectors accumulated in the feature vector buffer queue.

[0065] S1054. When the number of audio feature vectors accumulated in the feature vector buffer queue reaches a second preset threshold, multiple consecutive audio feature vectors that have reached the second preset threshold are extracted from the buffer queue as a feature sequence, and the feature sequence is input into the digital human driving model to generate a digital human image frame.

[0066] In step S1054, the feature sequence is input into the digital human driving model to generate a digital human image frame, specifically including: The feature sequence is input into the digital human driving model; within the digital human driving model, the input feature sequence is analyzed frame by frame to extract temporal change parameters related to the digital human's facial expressions, lip movements, and head posture; based on the temporal change parameters, the preset digital human 3D model or digital human 2D image template is driven to perform corresponding deformations and movements in the digital human driving model; the digital human 3D model after deformation and movement is rendered, or the digital human 2D image template is synthesized to finally generate a digital human image frame with corresponding visual movements.

[0067] The second preset threshold is set to 8 frames of audio feature vectors. This choice is made after comprehensively considering the smoothness and real-time performance of digital human movements: too few vectors will lead to fragmented movements, while too many will increase latency. The time length corresponding to 8 frames of vectors matches the temporal rhythm of the audio segmentation and feature extraction mentioned earlier, ensuring the synchronization between audio and visual movements. The digital human driving model is a technical framework that can map audio features to digital human visual movement parameters, used to establish the association between audio information and digital human facial and limb movements. Temporal variation parameters are the core data describing the dynamic changes of digital human movements, including facial expression intensity, mouth opening and closing angle, head rotation angle, etc., which directly determine the visual presentation effect of digital human image frames.

[0068] S1055. After generating the digital human image frame, update the feature vector buffer queue to continue caching subsequently generated audio feature vectors.

[0069] The feature vector buffer queue is updated using a sliding queue mechanism: the first 6 of the 8 audio feature vectors that have been formed into a feature sequence and input into the model are removed, and the last 2 vectors are retained to overlap with the newly added vectors. This design can avoid the discontinuity of the action in two adjacent frames of digital human images, ensure the continuity of the time sequence, and is consistent with the update logic of the sample collection window mentioned above, forming a closed loop of streaming processing.

[0070] In one specific implementation, such as Figure 2 As shown, a closed-loop process of feature caching, quantity monitoring, sequence extraction, model-driven, and queue updating is used to realize the temporal generation of audio features into digital human image frames. The overall process is as follows: First, a buffer queue is set up to cache feature vectors. The continuously generated audio feature vectors are enqueued in sequence. The number of vectors in the queue is monitored in real time. When the second preset threshold of 8 frames is reached, the continuous vectors are extracted to form a feature sequence and input into the digital human driving model. After generating a digital human image frame, the queue is updated to continuously receive subsequent feature vectors.

[0071] As an example, let's take the real-time processing of audio feature vectors corresponding to viewer comments by a virtual anchor as an example for detailed explanation: First, a feature vector buffer queue that supports ordered caching is set up in step S1051. The core function of the queue is to maintain the generation order of audio feature vectors and provide temporally consistent input for subsequent action generation.

[0072] Secondly, through step S1052, the audio feature vectors generated in step S104 are added to the buffer queue in the order of generation, for example, the audio feature vectors of frame 1, frame 2... frame 8 are enqueued in sequence.

[0073] Next, the number of vectors in the queue is monitored in real time through step S1053. When the 8th frame vector is enqueued, the number of vectors in the queue reaches the second preset threshold of 8 frames.

[0074] Subsequently, these 8 consecutive feature vectors are extracted as a feature sequence in step S1054 and input into the digital human driving model. The training process of the digital human driving model is as follows: We utilize a publicly available audio feature-digital human motion parameter pairing dataset, containing facial expression, lip-reading, and head pose annotations corresponding to different speech and tone. The model architecture employs an LSTM temporal feature extraction network and a fully connected parameter prediction network: the LSTM layer captures the temporal correlation of feature sequences, and the fully connected layer maps the temporal features to digital human motion parameters. The training objective is to minimize the mean squared error between the model's predicted temporal variation parameters and the true parameters in the dataset. The loss function formula is as follows: (3) In the formula, L is the loss value, M is the number of training samples, and K is the dimension of the action parameters of a single sample, such as 5 dimensions for facial expressions, 10 dimensions for lip movements, and 3 dimensions for head pose, for a total of 18 dimensions. Let j be the true action parameters of the i-th sample. Let be the j-th dimension action parameters predicted by the model for the i-th sample. The model is trained iteratively using the stochastic gradient descent algorithm until the loss value converges to a preset threshold, thus completing model training.

[0075] After receiving the feature sequence, the model first analyzes the 8-frame feature sequence frame by frame through the LSTM layer to capture the changing trends of pitch and rhythm and extract 18-dimensional temporal change parameters. Then, based on these parameters, it drives the preset digital human 3D model to control the model's facial muscle deformation, lip opening and closing movements, and head rotation. Finally, through image rendering techniques such as lighting rendering and texture mapping, the 3D model's action state is transformed into a 2D image, generating a digital human image frame corresponding to the audio features. For example, the audio features corresponding to the bullet screen text "The anchor is awesome" generate an image frame of "smiling + slight nodding + mouth closing".

[0076] Finally, the feature vector buffer queue is updated in step S1055, removing the feature vectors used in the first 6 frames and retaining the vectors from the last 2 frames. After the feature vector from the 9th frame is enqueued, the queue contains vectors from 3 frames, and the above monitoring, extraction, and generation process is repeated. The above example is merely one illustration of this application. In practical applications, the model structure and motion parameter dimensions can be adjusted according to the digital human image and the required level of motion detail; this application does not impose any limitations on this.

[0077] In another specific implementation, a digital human-driven model based on the Transformer architecture can be used, while a 2D image template of the digital human can be selected instead of a 3D model to adapt to lightweight deployment scenarios. During model training, a self-attention mechanism is used to capture the long-term dependencies of feature sequences, improving the prediction accuracy of time-varying parameters; the training dataset is further annotated with action parameters corresponding to different speech rates and dialects to enhance the model's adaptability.

[0078] The specific generation process is as follows: After the feature sequence is input into the model, the Transformer encoder focuses on key changes in the feature sequence through a multi-head attention layer, such as sudden changes in speech rate and pitch increase, to extract more accurate temporal change parameters; then, it drives a 2D image template, and adjusts the position of facial features in the template's facial region through image deformation algorithms such as thin-plate spline interpolation to achieve changes in mouth shape and expression; finally, through layer synthesis technology, the deformed 2D template is fused with the background image to generate a digital human image frame. The queue update mechanism remains unchanged, still ensuring action continuity by retaining some overlapping vectors. This implementation does not require 3D model rendering, has lower computational load and lower latency, and is suitable for lightweight application scenarios such as mobile and web applications.

[0079] This application achieves a precise mapping from audio features to digital human visual movements. The generated digital human image frames have smooth and natural movements and strong temporal consistency, which not only ensures the real-time performance of streaming processing, but also enables the digital human's movements to be precisely matched with the audio rhythm, thus meeting the visual presentation requirements of real-time interactive scenarios.

[0080] S106. The digital human image frame and the corresponding audio segment are output synchronously to form a digital human video.

[0081] The core objective of this step is to achieve precise synchronization between audio and visuals, integrating discrete image frames and audio segments into a complete and smooth digital human video, thus completing the final conversion from streaming text to digital human video.

[0082] In one specific implementation, the entire audio segment of the feature sequence corresponding to a single digital human image frame is first generated by tracing back the feature sequence. For example, a feature sequence consists of 8 audio feature vectors, each vector corresponding to 11,040 audio data samples and several audio segments of 800 samples each. These segments are integrated to obtain the target audio segment, and a one-to-one correspondence between the image frame and the audio segment is established to ensure that the action of the image frame matches the audio content.

[0083] Next, each digital human image frame and its corresponding target audio segment are assigned the same timestamp, with the timestamp value determined based on the actual playback duration of the audio segment. For example, if a target audio segment contains 11,040 samples, corresponding to a duration of 690 milliseconds at a 16kHz sampling rate, then the timestamp is marked as the start playback time of that segment. For instance, if the first image frame corresponds to an audio segment of 0-690 milliseconds, the timestamp is uniformly marked as 0 milliseconds.

[0084] Then, all the digital human image frames with timestamps are arranged in chronological order to form a continuous sequence of digital human image frames, such as 0 milliseconds, 690 milliseconds, 1380 milliseconds, etc. At the same time, the corresponding target audio segments are spliced ​​together in the same timestamp order to form a complete audio data stream, ensuring the continuity of audio playback.

[0085] Finally, the image frame sequence and audio data stream are synchronously synthesized based on the timestamp. During playback, the image frame and audio content at the corresponding time are matched according to the timestamp. For example, when the playback reaches 690 milliseconds, the second image frame and the corresponding audio segment from 690 to 1380 milliseconds are presented synchronously, and finally a smooth digital human video is generated and output.

[0086] The following is a complete example for steps 101-105, adapted for scenarios where a virtual anchor broadcasts real-time viewer comments. Details are as follows: It receives streaming text data in real time, such as the bullet screen text sent by the audience, "Hello everyone, today's live stream is full of useful information, remember to like and follow!" The text is transmitted in real time in segments according to sentences. The edge-tts engine is called to convert each segment of streaming text in turn to generate the first audio data in MP3 format. The conversion process is carried out synchronously with the text transmission, without waiting for the complete text.

[0087] Secondly, the generated MP3 format first audio data is converted into a unified 16-bit PCM format, 16kHz sampling rate, mono second audio data to eliminate the differences in the original audio format and adapt to the subsequent segmentation and feature extraction requirements.

[0088] Next, the second audio data is split into segments of a preset size, with 800 samples per segment, corresponding to an audio duration of 50 milliseconds at a sampling rate of 16kHz. Finally, several ordered audio segments are generated and output sequentially according to the text transmission and audio conversion order.

[0089] Then, a sliding window method is used to extract features from each audio segment. First, a sample collection window is set up, and 800 samples from each audio segment are added to the window sequentially, with the total number of samples in the window being counted in real time. When the number of samples accumulates to 11,040, corresponding to an audio duration of 690 milliseconds, the MFCC algorithm is used to calculate the features of the samples in the window, generating one frame of audio feature vector. After the feature vector is generated, the window is updated to retain the last 480 samples, and the system continues to receive samples from subsequent audio segments, performing feature extraction in a loop.

[0090] Next, a feature vector buffer queue is set up, and the audio feature vectors generated by S104 for each frame are added to the queue in sequence, with the number of vectors in the queue monitored in real time. When the number of vectors accumulates to 8 frames, these 8 consecutive frames of vectors are extracted as a feature sequence and input into the LSTM architecture digital human driving model. The model analyzes the temporal variation parameters of the feature sequence generation, drives the preset digital human 3D model to perform corresponding actions, and generates a digital human image frame through image rendering, such as a smiling expression matching the tone of the text, lip movements corresponding to pronunciation, and slight nodding movements. After the image frame is generated, the queue is updated to retain the last 2 frames of vectors and continues to cache subsequent feature vectors.

[0091] Finally, the target audio segment corresponding to each digital human image frame is determined. This target audio segment consists of all audio segments corresponding to the feature sequence generated for that image frame, with the starting playback time of the audio segment marked as a timestamp consistent with the timestamp assigned to the image frame and the target audio segment. All timestamped image frames are arranged sequentially to form an image frame sequence, and the corresponding target audio segments are concatenated into an audio data stream according to their timestamps. The two are synchronously synthesized based on their timestamps to generate and output the digital human video, realizing the complete process from bullet screen text to real-time digital human broadcasting.

[0092] In this embodiment, through full-process streaming processing and sliding window and buffer queue mechanisms, the end-to-end latency can be controlled within 200 milliseconds. The generated digital human video has smooth movements and synchronized audio and video, which can accurately adapt to the needs of real-time interactive scenarios of virtual anchors.

[0093] Figure 3 This is a schematic diagram illustrating a specific implementation of a real-time streaming text-to-digital human conversion system provided in this application, with reference to... Figure 3 The system may include: Text conversion module 31 is used to convert real-time received streaming text data into first audio data through the TTS engine; Format conversion module 32 is used to perform format conversion processing on the first audio data to generate second audio data; The audio segmentation module 33 is used to segment the second audio data according to a preset segmentation size to generate multiple audio segments; Feature extraction module 34 is used to extract audio features from each audio segment and generate an audio feature vector; Image generation module 35 is used to generate a digital human image frame corresponding to the feature sequence formed by accumulating the audio feature vector in time sequence. The synchronous output module 36 is used to synchronously output the digital human image frame and the corresponding audio segment to form a digital human video.

[0094] The real-time streaming text-to-digital human conversion system of this application is used to implement the aforementioned real-time streaming text-to-digital human conversion method. Therefore, the specific implementation of the real-time streaming text-to-digital human conversion system can be found in the embodiment section of the real-time streaming text-to-digital human conversion method above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0095] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described real-time conversion methods for streaming text to digital human.

[0096] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described methods for real-time conversion of streaming text to digital human.

[0097] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0098] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the real-time conversion method from streaming text to digital human.

[0099] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0100] The above provides a detailed description of the real-time conversion method and system for streaming text to digital human provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A real-time conversion method for streaming text to digital human, characterized in that, include: The real-time received streaming text data is converted into first audio data through the TTS engine; The first audio data is converted to generate the second audio data. The second audio data is divided into segments according to a preset segment size to generate multiple audio segments; For each audio segment, audio features are extracted to generate an audio feature vector; Based on the feature sequence formed by accumulating the audio feature vectors in chronological order, a digital human image frame corresponding to the feature sequence is generated; The digital human image frames and corresponding audio segments are output synchronously to form a digital human video.

2. The method according to claim 1, characterized in that, The step of generating a digital human image frame corresponding to the feature sequence formed by accumulating the audio feature vectors in chronological order includes: A feature vector buffer queue is set up, which is used to cache the audio feature vectors sequentially according to the generation order; Each audio feature vector is added sequentially to the feature vector buffer queue; Monitor the number of audio feature vectors accumulated in the feature vector buffer queue; When the number of audio feature vectors accumulated in the feature vector buffer queue reaches a second preset threshold, multiple consecutive audio feature vectors that have reached the second preset threshold are extracted from the buffer queue as a feature sequence, and the feature sequence is input into the digital human driving model to generate a digital human image frame. The second preset threshold is 8 frames of audio feature vectors. After generating the digital human image frame, the feature vector buffer queue is updated to continue caching subsequently generated audio feature vectors.

3. The method according to claim 2, characterized in that, The step of inputting the feature sequence into the digital human driving model to generate a digital human image frame includes: The feature sequence is input into the digital human driving model; Within the digital human driving model, frame-by-frame feature analysis is performed on the input feature sequence to extract temporal variation parameters related to the digital human's facial expressions, lip movements, and head posture. Based on the time-series change parameters, in the digital human driving model, the preset digital human 3D model or digital human 2D image template is driven to perform corresponding deformations and movements; The three-dimensional model of the digital human after deformation and motion is rendered, or the two-dimensional image template of the digital human is synthesized to finally generate a digital human image frame with corresponding visual actions.

4. The method according to claim 1, characterized in that, The step of extracting audio features from each audio segment to generate an audio feature vector includes: A sample collection window is set up, which is used to collect the samples contained in the audio segment in the order of reception; The samples contained in each of the audio segments are added sequentially to the sample collection window; The total number of samples accumulated in the sample collection window is determined in real time; When the total number of samples accumulated in the sample collection window reaches a first preset threshold, feature calculation processing is performed on all samples accumulated in the current window to generate an audio feature vector. The first preset threshold is 11040 samples. After generating the audio feature vector, the sample collection window is updated to continue collecting samples from subsequent audio segments.

5. The method according to claim 1, characterized in that, The audio segment size is 800 samples, corresponding to an audio duration of 50 milliseconds at a sampling rate of 16kHz.

6. The method according to claim 1, characterized in that, The second audio data is in 16-bit PCM format, with a sampling frequency of 16kHz and is mono.

7. The method according to claim 1, characterized in that, The TTS engine is edge-tts.

8. A real-time conversion system for streaming text to digital human, characterized in that, include: The text conversion module is used to convert real-time received streaming text data into first audio data through the TTS engine; The format conversion module is used to perform format conversion processing on the first audio data to generate the second audio data; The audio segmentation module is used to segment the second audio data according to a preset segmentation size to generate multiple audio segments; The feature extraction module is used to extract audio features from each audio segment and generate an audio feature vector. The image generation module is used to generate a digital human image frame corresponding to the feature sequence formed by accumulating the audio feature vector in chronological order. The synchronous output module is used to synchronously output the digital human image frame and the corresponding audio segment to form a digital human video.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the real-time conversion method for streaming text to digital human as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the real-time conversion method from streaming text to digital human as described in any one of claims 1 to 7.

Images