Text-driven end-to-end AI virtual digital human generation method and system

Abstract

The invention discloses an end-to-end AI virtual digital human text broadcast video generation method, which comprises the following steps: configuring the tone and speed of a speaker and the spoken language connection, laughter, pause and emotional degree of audio, converting text content into a corresponding audio file, and carrying out alignment operation on the audio file and the text content; extracting a human face frame to obtain potential features of the human face, and splicing all image frames containing the human face and the potential features of the human face in an inverted order to obtain image features; obtaining coding features of the audio, and obtaining audio features corresponding to each frame in combination with the frame rate of the audio; splicing the audio features and the image features of each frame, predicting and decoding potential features, and obtaining a face picture of each frame after prediction; and covering each frame of face picture after prediction back to the original frame, combining the audio and the video, adding the generated subtitles, and generating a final AI virtual digital human text broadcast video. According to the invention, a real-time high-quality digital human image can be obtained.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer graphics, and in particular to a text-driven end-to-end AI virtual human generation method and system. Background Art

[0002] Digital humans are virtual human images created using computer digitization technology. These images can have vivid appearances, intelligent behaviors, and realistic movements. The development of digital human technology has gone through multiple stages: from the initial use of 3D modeling, based on hand-made or physical simulation methods to build digital human models; then, with the development of animation technology, the use of skeletal animation, skinning, and other technologies to make digital human movements and expressions more realistic; then, with the introduction of motion capture technology, digital humans can more accurately simulate real-life movements; finally, with the continuous development of artificial intelligence, computer graphics, and computing power in recent years, deep learning technology can better simulate human language, appearance, and behavior, allowing the creation of more realistic virtual characters.

[0003] Compared to traditional methods, the use of deep learning technology enables digital humans to automatically learn and adapt to new data and situations, reducing the need for human intervention and improving the autonomy and intelligence of digital human technology. Furthermore, through hardware acceleration, high-quality digital human performances can be generated in a shorter time, making them suitable for scenarios requiring real-time interaction and rendering. A common related technology utilizes AI to synchronize the mouth shapes of video characters with the input sound. Some early lip synchronization technologies had limitations, such as being optimized for specific languages ​​and lacking support for other languages; slow processing speeds that could not meet the requirements of real-time synchronization; and a lack of user customization options that prevented them from adapting well to different speaking styles, speech rates, or emotional expressions.

[0004] Speech generation technology refers to the process of automatically generating audio from text, speech, or other sources using machine learning and artificial intelligence techniques. This technology makes it easier for computer systems to interact with users, providing a more natural and convenient way to communicate. It is widely used in fields such as virtual assistants and chatbots, accessible communications, language learning, media, and entertainment. Previous speech generation technologies can produce speech that lacks naturalness and fluency, and can have a monotonous emotional expression. Furthermore, their processing capabilities for long texts are limited, and word segmentation issues can occur, affecting the quality of the synthesized speech. Summary of the Invention

[0005] The purpose of this invention is to propose a text-driven end-to-end AI virtual digital human generation method and system to remove the model's support restrictions on long texts, improve the model's adaptability to texts containing a variety of different character types, provide support for the broadcast subtitle function, and achieve a more natural, realistic, and real-time responsive text-to-digital human generation route.

[0006] The technical solution to achieve the purpose of the present invention is: an end-to-end AI virtual digital human text broadcast video generation method, comprising the following steps:

[0007] Step 1: Preprocessing of broadcast text

[0008] Preprocessing user input text, including segmenting long text, replacing symbols that the generative speech model cannot recognize, and adding modifiers based on the symbol type; converting numeric characters that the generative speech model cannot recognize to make them more consistent with daily reading and writing habits;

[0009] Step 2: Announce text-to-speech

[0010] Configure the speaker's timbre and speed, as well as the audio's spoken connection, laughter, pauses, and emotional level, and use a generative speech model to convert the pre-processed text content into corresponding audio files;

[0011] Step 3: Generate subtitle file

[0012] Use the forced alignment method to align the generated audio with the text content, so as to obtain the timestamp information in the audio corresponding to each text character and create a subtitle file;

[0013] Step 4: Video processing and feature extraction

[0014] Input video or image as the basic image of the digital human, extract the face frame from the input video or image, convert the face into a representation in the latent space, obtain the latent features of the face, and reverse the order of all image frames containing the face and the latent features of the face and then splice them together to obtain the image features;

[0015] Step 5: Audio Feature Extraction

[0016] Use the audio encoder Whisper to obtain the audio encoding features of the audio file, and combine it with the audio frame rate to obtain the audio features corresponding to each frame;

[0017] Step 6: Integrate audio and video features

[0018] Concatenate the audio and image features of each frame, use UNet to predict, obtain the predicted latent features, and decode the latent features to obtain the predicted face image of each frame;

[0019] Step 7: Audio, video and subtitle synthesis

[0020] Each frame of the predicted face image is overwritten back to the position of the face frame extracted in the original frame, and then the multimedia processing software ffmpeg is used to merge the audio and video, and the generated subtitles are added to generate the final AI virtual digital human text broadcast video.

[0021] Furthermore, step 1: pre-processing of the broadcast text, the specific method is as follows:

[0022] Step 1.1: Segmentation and modification of broadcast text

[0023] Use periods as separators to split the original text into short sentences, then reconnect the sentences into a section. Keep the length of the text in the section smaller than 130 characters, and the corresponding generated audio duration within 30 seconds. If the length exceeds this limit, it will be considered as the next section.

[0024] Reconnect short sentences and replace symbols in the text, including: replacing colons, semicolons, and colons with commas; and replacing question marks, exclamation marks, and ellipsis marks with periods;

[0025] Add a pause modifier "[uv_break]" after the period, and use each modified section as a unit for audio generation;

[0026] Step 1.2: Digital Conversion

[0027] Converting numeric characters into Chinese characters includes two different conversion methods: directly mapping numeric characters into corresponding Chinese characters; and reflecting numeric units in the reading according to the reading habits of numbers in Chinese.

[0028] Further, step 2: broadcast text to speech, wherein:

[0029] The params_infer_code variable is used to globally control the speaker's timbre and speed. The spk_emb parameter controls the speaker's main timbre; the speed parameter controls the overall audio speed.

[0030] The params_refine_text variable is used to globally control the oral connection, laughter, pause level, and emotional level in the speech. The parameter oral controls the degree of colloquialism in the text; the parameter laugh controls the degree of laughter in the audio; the parameter break controls the degree of pause in the audio; and the parameter temperature controls the emotional volatility of the audio.

[0031] Further, step 3: subtitle file generation, the specific method is:

[0032] Input the audio into the Wav2Vec2 model, and the output is an intermediate tensor, which represents the probability distribution of the token corresponding to each frame of the audio;

[0033] Normalize the preprocessed text into phonetic letters and convert it into the corresponding Token sequence through Tokenizer;

[0034] The Wav2Vec2 model and Tokenizer use the same set of Token comparison lists to align the intermediate tensor output by Wav2Vec for the audio with the Token sequence output by the Tokenizer for the standardized text, that is, to obtain the timestamp and score corresponding to each Token in the Token sequence. Finally, the timestamp information in the audio corresponding to each text character is combined into an SRT format subtitle file for output.

[0035] Furthermore, step 4: video processing and feature extraction, the specific method is:

[0036] If the input video is used as the basic image of the digital human, the video is first decomposed into video frames, the face frame of each frame is obtained, and the face is cropped frame by frame; if the input image is used as the basic image of the digital human, the face frame of the image is directly obtained and the face in the image is cropped;

[0037] The cropped face image is converted into a latent space representation by the encoder part of the variational autoencoder to obtain the latent features of the face;

[0038] All image frames containing faces and potential features of faces are reversed and then spliced ​​together to obtain image features.

[0039] Further, step 5: audio feature extraction, where:

[0040] The input audio is first converted into Whisper encoding features, and then the audio features corresponding to each frame are obtained according to the audio frame rate, expressed as the dimensions of (frames, a, b), where the first dimension frames represents the number of audio frames, and a and b are the dimensions specified by the whisper model output features.

[0041] Furthermore, step 6: integrating audio and video features, wherein:

[0042] UNet uses the UNet2DConditionModel model.

[0043] An end-to-end AI virtual digital human text broadcast video generation system implements the end-to-end AI virtual digital human text broadcast video generation method to achieve end-to-end AI virtual digital human text broadcast video generation, including a broadcast text preprocessing module, a broadcast text to speech module, a subtitle file generation module, a video processing and feature extraction module, an audio feature extraction module, an audio and video feature integration module and an audio, video and subtitle synthesis module.

[0044] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the end-to-end AI virtual digital human text broadcast video generation method is implemented to achieve end-to-end AI virtual digital human text broadcast video generation.

[0045] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the end-to-end AI virtual digital human text broadcast video generation method is implemented to achieve end-to-end AI virtual digital human text broadcast video generation.

[0046] Compared with the existing technology, the present invention has the following significant advantages: it creates high-quality, multilingual, and highly realistic digital human images through steps such as text preprocessing, speech synthesis, and lip alignment, improves the flexibility of the model and the generation effect of complex and changeable text content, removes the limitations of long text in audio generation, and increases the processing and optimization of characters; it also combines the optimized audio generation model with advanced lip synchronization technology to obtain real-time, high-quality digital human images. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the end-to-end AI virtual digital human generation solution driven by text.

[0048] Figure 2 It is a list of optional parameters for global settings of sentences.

[0049] Figure 3 This is the Loss function principle of the lip synchronization task. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] An end-to-end AI virtual human text broadcast video generation method includes the following steps:

[0052] Step 1: Preprocessing of broadcast text

[0053] Preprocess the original text to improve the quality of the generated audio. This includes segmenting long texts, replacing symbols that the speech generation model can't recognize, and adding modifiers based on the symbol type. Furthermore, converting numeric characters that the speech generation model can't recognize to make them more consistent with daily reading and writing habits.

[0054] Step 1.1: Segmentation and modification of broadcast text

[0055] Use punctuation marks (such as periods, commas, question marks, exclamation marks, commas, semicolons, colons, etc.) as separators to split the original text into short sentences, and reconnect the split short sentences into a section. The length of the text in the connected section is controlled to be less than 130 characters (corresponding to 30 seconds of generated audio length). If the length exceeds this limit, it will be regarded as the next section.

[0056] While reconnecting short sentences, we replace symbols in the text: commas are replaced with commas, and question marks, exclamation marks, and ellipsis marks are replaced with periods. This addresses the model's filtering of specific symbols and maintains the tone and pauses of the generated audio. A pause modifier, "[uv_break]," is added after the period to better adjust the rhythm, express emotion, and emphasize specific parts.

[0057] Each modified section is treated as a unit for audio generation. This allows the audio output of the entire text to be generated by simply concatenating the audio tensors generated for each section. This processing method not only overcomes the model's limitations on supporting long texts but also addresses the issue of timbre discrepancies caused by the model processing short texts. This increases the output audio length while maintaining maximum stability.

[0058] Step 1.2: Digital Conversion

[0059] After step 1.1, the original text has been basically processed. However, the generative model cannot recognize various numeric characters in the text. Therefore, these numeric characters need to be converted to Chinese characters to ensure that the corresponding characters in the generated audio are pronounced correctly. Two different conversion methods are provided: directly mapping numeric characters to their corresponding Chinese characters; and reflecting the numeric units in the pronunciation according to the Chinese pronunciation conventions for numbers. By default, the system automatically assigns a numeric character processing method based on the context of the text. Users can also specify the corresponding numeric conversion method as needed.

[0060] Step 2: Announce text-to-speech

[0061] Use the generative speech model to convert the text preprocessed in step 1 into audio.

[0062] Create a generative speech model object and load pre-trained models, including generative pre-trained models based on the attention mechanism and Transformer architecture; discrete variational autoencoders are used to learn data representations.

[0063] Make global settings for sentences, control the speaker's voice and speed, as well as the audio's spoken connection, laughter, pauses, and emotional level. Optional parameter settings include Figure 2 The params_infer_code variable primarily controls the speaker's timbre and speed globally: the spk_emb parameter controls the speaker's primary timbre, and the speed parameter controls the overall speed of the audio. The params_refine_text variable primarily controls the spoken connection, laughter, pause intensity, and emotionality of the speech: the oral parameter controls the colloquialism of the text; the laugh parameter controls the amount of laughter in the audio; the break parameter controls the amount of pauses in the audio; and the temperature parameter controls the emotional volatility of the audio.

[0064] The generative speech model uses the set parameters to infer the text optimized in step 1, and exports it as an audio file after inference is completed.

[0065] Step 3: Generate subtitle file

[0066] To add captions to the digital human avatar, we need to generate a caption file corresponding to the audio. Since the generative speech model in step 2 doesn't provide timestamp information for the intervals between words in the audio corresponding to each character in the text, we use a forced alignment method to align the text with the generated audio.

[0067] The audio is input into the Wav2Vec2 model, and the output is an intermediate tensor, which represents the probability distribution of the tokens corresponding to each frame of the audio, that is, the probability distribution of each frame in the audio corresponding to each token.

[0068] When processing Chinese, you only need to standardize Chinese into pinyin letters, and then convert the standardized text into the corresponding Token sequence through Tokenizer.

[0069] By aligning the tensor with the token sequence, we can obtain the timestamp and score corresponding to each token in the token sequence. When processing Chinese text, we can normalize the text into phonetic alphabets. Then, we use the normalized phonetic alphabet text to perform the above alignment operation with the audio. This will obtain the timestamp information corresponding to each phonetic alphabet in the text, thus determining the temporal relationship between the texts. Finally, we combine this temporal relationship to produce an SRT-formatted subtitle file for output.

[0070] Note that the same token mapping list should be used for the Wav2Vec2 model and Tokenizer used here. The token mapping list maps all English letters to numbers. In addition to the twenty-six English letters, it also maps the hyphen "-", single quote "'", and asterisk "*", forming a mapping relationship of a total of 29 characters.

[0071] For example, here the text "My name is Li Bing" is aligned with the corresponding audio. The xpinyin toolkit is used to standardize the original Chinese text into pinyin letters "wojiaolibing", and then these pinyin letters are transformed into a token sequence through the Tokenizer: (19, 5, 22, 2, 1, 5, 12, 2, 17, 2, 4, 14), where each number corresponds to an English letter. The original audio is transformed into an intermediate tensor through the Wav2Vec2 model, and the dimension of the intermediate tensor is (1, 77, 29). The model divides the original audio into 77 frames, and each frame has a probability distribution for the 29-character mapping relationship of the token list. Through a simple alignment operation, the frame situation corresponding to each token can be obtained. For example, the token corresponding to the third letter "j" in the pinyin is "22", and the starting frame of this letter in the original audio is the 28th frame and the ending frame is the 29th frame. Dividing the frame correspondence by the sampling rate of the audio can obtain the time where the corresponding letter is located in the audio. Finally, it is sorted into the format of an SRT subtitle file for output.

[0072] Step 4: Video Processing and Feature Extraction

[0073] The key to creating a digital human lies in lip synchronization with the audio. As a multi-modal task, lip synchronization requires integrating the information of audio and video frames. It is required to use face frames and synchronized audio frames as inputs, and each obtains potential features through an encoder. The input video or image provides the basic image of the finally generated digital human. For video data, it needs to go through a preprocessing and corresponding feature extraction process to be transformed into the corresponding latent space vector. For image data, its corresponding latent space vector is directly obtained through feature extraction.

[0074] Step 4.1: Video Preprocessing

[0075] The video first needs to be decomposed into the form of image frames to facilitate feature extraction for each frame of the image. At the same time, a single picture or multiple pictures can also be directly used as inputs.

[0076] Step 4.2: Image Feature Extraction

[0077] To obtain the face frame position for each decomposed image frame, the full-body pose estimator is first used to locate the key points of the human face in the image. A pixel value range (box_shift) is then defined to shift the face frame boundary upward or downward to expand or shrink the face frame. Finally, this shift value is combined with the obtained facial key points to obtain the face frame position. Based on the obtained face frame position, the face is cropped frame by frame. The cropped face image is passed through the encoder portion of the variational autoencoder to obtain the corresponding latent features.

[0078] The variational autoencoder is a typical "encoder-decoder" architecture. It first fits the data into a latent space to obtain latent features. The input data is converted from the format of (1, 3, 256, 256) to the latent features of the dimension (1, 4, 32, 32) after passing through the encoder. The decoder acts as a generator, responsible for restoring the learned latent representation to the original data. Image generation is achieved by using a variational autoencoder, and the generated image is controlled by modifying the latent feature vector.

[0079] All image frames containing faces and the latent features output by the variational autoencoder are concatenated with their respective inverse results to achieve smoothing.

[0080] Step 5: Audio Feature Extraction

[0081] Audio is encoded using the existing Whisper model to obtain its corresponding features. The input audio is first converted to Whisper encoding features, and then the Whisper encoding features corresponding to each frame are obtained based on the audio frame rate. The processed audio becomes a dimension (frames, 5, 384), where the first dimension, frames, represents the number of audio frames.

[0082] Whisper is a general-purpose speech recognition model trained on a wide range of speech datasets. It can perform tasks such as multilingual speech recognition and speech translation. By using the Whisper audio encoder, it extracts high-dimensional features from the audio, eliminating differences in timbre and pronunciation between different speakers and improving the model's robustness.

[0083] Step 6: Integrate audio and video features

[0084] This project uses a Uniform Resource Planning (UNet) to perform a self-attentional integration of processed audio and video frame features. Specifically, the UNet2DConditionModel model is used. Compared to the original UNet, this model includes an intermediate input embedding layer. This embedding layer integrates the latent image features obtained in step 4 with the audio features obtained in step 5, ensuring that the predicted face includes audio information. Specifically, the face's mouth shape moves in accordance with the audio. The model also uses an attention mechanism for this integration, allowing it to better weight the influence of audio features on the target outcome.

[0085] The UNet network is commonly used for image translation and image segmentation tasks. Its distinguishing feature is its U-shaped structure, which allows the model to analyze data layer by layer from different perspectives. Furthermore, the symmetry of the UNet allows it to translate latent features while ensuring structural consistency and integrity as data enters and leaves the network.

[0086] The UNet output is the predicted latent features, which contain the features of the face frame corresponding to the sound and mouth shape. The predicted face image is then obtained through the decoder part of the variational autoencoder.

[0087] Step 7: Audio, video and subtitle synthesis

[0088] Based on the facial box positions recorded in each frame in step 4.2, the predicted facial image is overlaid onto the corresponding position of the original image frame after reverse stitching in step 4.2. Finally, the multimedia processing software ffmpeg is used to merge the video frames and audio into a video, and the SRT subtitle file generated in step 3 is added to complete the digital human generation.

[0089] Example

[0090] In order to verify the effectiveness of the scheme of the present invention, the following experimental design was carried out.

[0091] After deploying the aforementioned text-based AI virtual human generation solution, users can enter the text they want the digital human to narrate and select the digital human's voice and image. The system then generates the digital human's image and displays subtitles in real time based on the user's input. Users can continuously enter new text, and the digital human will respond in real time. For example, for user A, the system receives the text and user-defined image and generates the digital human in real time as follows:

[0092] Step 1: User A first inputs the text that they want the system-generated digital person to speak. After receiving the text input by the user, the system optimizes and preprocesses the original text content. Specifically, it can be divided into: segmenting the text with periods as intervals; recombining the segmented text into sections; replacing the original commas, semicolons, and colons in the text with commas, and replacing question marks, exclamation marks, and ellipsis marks with periods; adding a pause modifier after the period; and adaptively replacing the numeric characters in the text with Chinese characters that match the context. Through preprocessing operations, the text removes the interference of special symbols, maintains a consistent format and style, and improves the quality and consistency of the generated speech.

[0093] Step 2: User A customizes the speech generation interface by selecting the target audio's timbre, speaking rate, and emotional characteristics. The system then creates a speech generation model object and loads the user-defined parameters to generate speech from the optimized text.

[0094] Step 3: Create an SRT subtitle file based on the original text and the audio generated in the second step. Use the xpinyin tool to standardize the original text into pinyin letters, and convert the pinyin letters into a token sequence through the Tokenizer. Use the Wav2Vec2 model to convert the audio into an intermediate tensor, which contains the probability distribution of each audio frame for all tokens. Align the token sequence with the intermediate tensor to obtain the number of audio frames corresponding to each token in the token sequence, and then divide it by the sampling rate to obtain the time information of the audio corresponding to each token (that is, the original text character). Organize the results and output them as an SRT format subtitle file.

[0095] Step 4: The user also needs to input an image or a video to determine the basic image of the digital human. If the user inputs the digital human image in the form of a video, the system will first process the input video into video frames, and then detect the face box containing the lips for each video frame and crop it. If the input is a picture, the face box containing the lips in the image will be directly detected and cropped. The cropped face frame is encoded using a variational autoencoder to obtain the corresponding latent features. The original image frame and the latent features are spliced ​​with the results of reversing themselves. The spliced ​​latent features are used for face prediction in step 6; the spliced ​​original image frame is used to predict the coverage of the face frame in step 7.

[0096] Step 5: Use Whisper encoder to obtain audio features for the speech generated in step 2.

[0097] Step 6: The concatenated image latent features from step 4 are fed into the UNet as input. The audio features obtained in step 5 are embedded into the UNet's intermediate embedding layer using the attention mechanism. The UNet performs inference to obtain the predicted latent features. The decoder of the variational autoencoder is then used to extract the corresponding face prediction frame.

[0098] Step 7: Overlay all predicted face frames onto the face frame corresponding to the original image frame spliced ​​in Step 4. Use ffmpeg to combine all predicted face frames with the audio, then add the SRT subtitle file generated in Step 3 to generate the final digital human broadcast video. User A can enter new text based on this, while keeping other parameters unchanged, and the system will update the digital human broadcast content in real time.

[0099] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An end-to-end AI virtual digital human text broadcast video generation method, characterized in that: The steps include: Step 1: Preprocessing of broadcast text Preprocessing user input text, including segmenting long text, replacing symbols that the generative speech model cannot recognize, and adding modifiers based on the symbol type; converting numeric characters that the generative speech model cannot recognize to make them more consistent with daily reading and writing habits; Step 2: Announce text-to-speech Configure the speaker's timbre and speed, as well as the audio's spoken connection, laughter, pauses, and emotional level, and use a generative speech model to convert the pre-processed text content into corresponding audio files; Step 3: Generate subtitle file Use the forced alignment method to align the generated audio with the text content, so as to obtain the timestamp information in the audio corresponding to each text character and create a subtitle file; Step 4: Video processing and feature extraction Input video or image as the basic image of the digital human, extract the face frame from the input video or image, convert the face into a representation in the latent space, obtain the latent features of the face, and reverse the order of all image frames containing the face and the latent features of the face and then splice them together to obtain the image features; Step 5: Audio Feature Extraction Use the audio encoder Whisper to obtain the audio encoding features of the audio file, and combine it with the audio frame rate to obtain the audio features corresponding to each frame; Step 6: Integrate audio and video features Concatenate the audio and image features of each frame, use UNet to predict, obtain the predicted latent features, and decode the latent features to obtain the predicted face image of each frame; Step 7: Audio, video and subtitle synthesis The face frame of each frame of the predicted face image is overwritten back to the position of the face frame extracted in the original frame, and then the multimedia processing software ffmpeg is used to merge the audio and video, and the generated subtitles are added to generate the final AI virtual digital human text broadcast video.

2. The end-to-end AI virtual digital human text broadcast video generation method according to claim 1 is characterized in that: Step 1: Preprocess the broadcast text. The specific method is as follows: Step 1.1: Segmentation and modification of broadcast text Use periods as separators to split the original text into short sentences, then reconnect the sentences into a section. Keep the length of the text in the section smaller than 130 characters, and the corresponding generated audio duration within 30 seconds. If the length exceeds this limit, it will be considered as the next section. Reconnect short sentences and replace symbols in the text, including: replacing colons, semicolons, and colons with commas; and replacing question marks, exclamation marks, and ellipsis marks with periods; Add a pause modifier "[uv_break]" after the period, and use each modified section as a unit for audio generation; Step 1.2: Digital Conversion Converting numeric characters into Chinese characters includes two different conversion methods: directly mapping numeric characters into corresponding Chinese characters; and reflecting numeric units in the reading according to the reading habits of numbers in Chinese.

3. The end-to-end AI virtual digital human text broadcast video generation method according to claim 1 is characterized in that: Step 2: Translate text to speech, where: The params_infer_code variable is used to globally control the speaker's timbre and speed. The spk_emb parameter controls the speaker's main timbre; the speed parameter controls the overall audio speed. The params_refine_text variable is used to globally control the oral connection, laughter, pause level, and emotional level in the speech. The parameter oral controls the degree of colloquialism in the text; the parameter laugh controls the degree of laughter in the audio; the parameter break controls the degree of pause in the audio; and the parameter temperature controls the emotional volatility of the audio.

4. The end-to-end AI virtual digital human text broadcast video generation method according to claim 1 is characterized in that: Step 3: Generate subtitle file. The specific method is as follows: Input the audio into the Wav2Vec2 model, and the output is an intermediate tensor, which represents the probability distribution of the token corresponding to each frame of the audio; Normalize the preprocessed text into phonetic letters and convert it into the corresponding Token sequence through Tokenizer; The Wav2Vec2 model and Tokenizer use the same set of Token comparison lists to align the intermediate tensor output by Wav2Vec for the audio with the Token sequence output by the Tokenizer for the standardized text, that is, to obtain the timestamp and score corresponding to each Token in the Token sequence. Finally, the timestamp information in the audio corresponding to each text character is combined into an SRT format subtitle file for output.

5. The end-to-end AI virtual digital human text broadcast video generation method according to claim 1 is characterized in that: Step 4: Video processing and feature extraction. The specific method is: If the input video is used as the basic image of the digital human, the video is first decomposed into video frames, the face frame of each frame is obtained, and the face is cropped frame by frame; if the input image is used as the basic image of the digital human, the face frame of the image is directly obtained and the face in the image is cropped; The cropped face image is converted into a latent space representation by the encoder part of the variational autoencoder to obtain the latent features of the face; All image frames containing faces and potential features of faces are reversed and then spliced ​​together to obtain image features.

6. The end-to-end AI virtual digital human text broadcast video generation method according to claim 1 is characterized in that: Step 5: Audio feature extraction, where: The input audio is first converted into Whisper encoding features, and then the audio features corresponding to each frame are obtained according to the audio frame rate, expressed as the dimensions of (frames, a, b), where the first dimension frames represents the number of audio frames, and a and b are the dimensions specified by the whisper model output features.

7. The end-to-end AI virtual digital human text broadcast video generation method according to claim 1 is characterized in that: Step 6: Integrate audio and video features, including: UNet uses the UNet2DConditionModel model.

8. An end-to-end AI virtual digital human text broadcast video generation system, characterized by: Implement the end-to-end AI virtual digital human text broadcast video generation method described in any one of claims 1-7 to realize end-to-end AI virtual digital human text broadcast video generation, including a broadcast text preprocessing module, a broadcast text to speech module, a subtitle file generation module, a video processing and feature extraction module, an audio feature extraction module, an audio and video feature integration module and an audio, video and subtitle synthesis module.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for generating an end-to-end AI virtual digital human text broadcast video according to any one of claims 1 to 6 is implemented to realize end-to-end generation of an AI virtual digital human text broadcast video.

10. A computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the end-to-end AI virtual digital human text broadcast video generation method described in any one of claims 1-6 is implemented to achieve end-to-end AI virtual digital human text broadcast video generation.

Images