A method for generating real-time conversational digital avatars with audiovisual perception capabilities

Abstract

This invention belongs to the field of image-speech generation, proposing a real-time dialogue-based digital avatar generation method with audiovisual perception capabilities. In the training and preprocessing stages, audio and video materials of a real-life prototype are recorded and preprocessed. These materials are then used to train a speech synthesis model, a lip-syncing model, and a facial enhancement model. In the digital avatar generation stage, a buffer is established, and through four steps—perception and response, lip-syncing synchronization, facial enhancement, and audio / video output—a real-time dialogue digital avatar is generated. This invention can synthesize high-fidelity digital avatars, enabling real-time interactive dialogue with users.

Description

Technical Field

[0001] This invention belongs to the field of image-speech generation, and in particular relates to a method for generating a real-time conversational digital avatar with audiovisual perception capabilities. Background Technology

[0002] Digital avatars refer to virtual human systems that closely resemble real-life counterparts and possess interactive capabilities. Real-time dialogue-based digital avatar generation involves a digital avatar sensing user input such as voice and gestures in a dialogue scenario, synthesizing real-time responses, and generating a sequence of realistic virtual human images with synchronized lip movements and semantically appropriate body language. This outputs audio and video content, enabling real-time dialogue between the digital avatar and a real person. Driven by advancements in artificial intelligence and virtual reality, digital avatar generation is gaining increasing attention and is widely applied in human-computer interaction, film and television production, virtual anchors, and intelligent employees. Real-time dialogue-based digital avatar technology with audiovisual perception capabilities focuses on user service functions, providing personalized services through contextual dialogue, thus possessing significant public benefit.

[0003] However, due to the complexity of facial structure, the diversity of lip movements, and the complexity of body movements, the generation of conversational digital avatars with perceptual capabilities has become one of the key and challenging research areas in the field of computer audiovisual perception. Currently, real-time conversational digital avatar generation suffers from the following problems: 1) Weak perceptual ability: Strong and accurate perceptual ability is the guarantee for a good interactive experience for digital avatars. Existing digital avatar solutions usually only have text perception or simple audiovisual perception. Single or inaccurate perceptual ability will seriously affect the interactive efficiency and experience of digital avatars. For example, audiovisual perception requires accurately locating the user's position in front of the camera and quickly recognizing the user's basic gestures. However, the randomness of the user's position and the diversity of gestures pose certain challenges to audiovisual perception. Existing simple audiovisual perception cannot accurately locate users and recognize actions; 2) Low image realism: The fidelity and clarity of the digital clone image are important factors affecting user experience. Existing digital clone solutions generally suffer from poor image clarity and low fidelity, reducing the realism of the image and greatly affecting user experience, thus limiting the service areas and applications of digital clones; 3) Monotonous and stiff gestures in digital clones: Gestures are natural actions that people produce when speaking. Gestures are more varied and vivid, but existing digital avatar generation solutions allow only a limited range of gestures, which can easily lose user appeal. Furthermore, the minimum time intervals between different gestures are relatively long, resulting in stiffness and a lack of naturalness. 4) Inconsistency between speech and lip movements: In digital avatar generation, inaccurate multimodal feature modeling and matching of audio and video can easily lead to a mismatch between lip movements and speech. 5) Poor real-time performance: Conversational digital avatars require a rapid response to user commands, including speech, gestures, and video images. This demands high real-time performance, but existing systems often suffer from high latency, resulting in a poor user experience. Summary of the Invention

[0004] This application provides a digital avatar generation method to address the problems of weak perception, limited gestures, inconsistency between speech and lip movements, and poor real-time performance in real-time dialogue-based digital avatar generation tasks. The technical solution adopted by this invention is as follows:

[0005] A method for generating real-time conversational digital avatars with audiovisual perception capabilities, characterized by the following steps:

[0006] During the training and preprocessing phases, audio and video materials are recorded and preprocessed, and then used to train the speech synthesis model, lip-shape generation model, and face enhancement model.

[0007] A buffer is established during the digital avatar generation stage. Through four steps—perception and response, lip-syncing, facial enhancement, and audio-visual output—a digital avatar for real-time dialogue is generated.

[0008] The method described above is characterized in that, during the training and preprocessing phase, audio and video materials are recorded and preprocessed, including:

[0009] Record audio and video materials of a real-life prototype digital avatar under the same hardware conditions and environment, including voice audio, driving video, motion video, and training audio and video. The voice audio is an audio clip of the real-life prototype reading a pre-prepared text. The driving video is a video clip of the real-life prototype in a natural state without large facial movements and with both hands in a designated position on the body. The motion video consists of multiple videos of the real-life prototype performing various hand gestures. The hand gestures will be used as preset actions when inserting actions. At the beginning and end of each action, the person's hands must remain in the designated position where the hands were held in the driving video to facilitate the transition with the driving video. The training audio and video consists of one or more audio and video clips of the real-life prototype reading aloud in a natural state.

[0010] Preprocessing includes:

[0011] Specify the video frame rate and adjust all video footage to that frame rate;

[0012] Using the existing frame interpolation model, the last frame of the driving video is used as the starting frame and the first frame of the driving video is used as the ending frame for frame interpolation. A set of composite frames synthesized by the frame interpolation model is obtained, which can make the beginning and end of the driving video transition naturally. These composite frames are then placed at the end of the driving video in chronological order. Specifically, assuming that the last frame of the driving video is the Nth frame, the k composite frames obtained by interpolating between the starting frame and the ending frame are regarded as the N+1 to N+k frames in sequence.

[0013] Define a set of trigger keywords for body movements, and determine the corresponding action videos for each keyword;

[0014] Define a set of action-response relationships to determine the text of the fixed response voice corresponding to various physical action interaction inputs received by the digital avatar system from the user.

[0015] The method described above is characterized in that, during the training and preprocessing phases, audio and video materials are used to train a speech synthesis model, a lip-shape generation model, and a face enhancement model, including:

[0016] The speech synthesis model is obtained by fine-tuning an existing speech synthesis model pre-trained on a publicly available large speech dataset using the speech audio and the text corpus. The input of the speech synthesis model is text, and the output is the audio corresponding to the text and the Mel spectrum of the audio.

[0017] The audio of the training audio-video is extracted and converted into Mel spectrogram features with the same frame rate as the video. The existing audio-driven lip-sync model, which has been pre-trained on a large amount of publicly available audio-video data, is fine-tuned using the RGB frames of the training audio-video and the Mel spectrogram features of the audio in the training audio-video to obtain the lip-sync generation model.

[0018] The Mel spectrum of the audio in the training audio-video and the driving video are input into the lip-shape generation model to obtain a low-resolution synthetic video synchronized with the audio in the training audio-video. The synthetic video and the video in the training audio-video are used as source data and target data, respectively, to train the existing face replacement network to obtain the face enhancement model.

[0019] The method described above is characterized by establishing a buffer during the digital clone generation stage, including:

[0020] Extract all RGB frames of the driving video and number them in ascending order of time;

[0021] A buffer is established with a limb movement video frame queue, a queue to be enhanced, an audio queue, a Mel spectrum queue, an audio tag queue, and an output queue. The limb movement video frame queue stores limb movement video frames, the queue to be enhanced stores video frames to be enhanced, the audio queue stores audio to be played, the Mel spectrum queue stores the Mel spectrum features of the audio to be played, the audio tag queue stores Boolean tags indicating whether audio should be played, and the output queue stores video frames that have been enhanced for the final output.

[0022] The method described above is characterized by the following specific process in the perception and response steps during the digital clone generation stage:

[0023] The system monitors and acquires real-time audio data, while the camera captures video data from the user. It determines the presence of voice interaction information in the audio data based on audio loudness, and identifies the presence and type of interactive actions in the video data based on existing human keypoint recognition models.

[0024] 1) If the aforementioned voice interaction information or interaction action exists:

[0025] Respond with text based on voice interaction information or interactive actions to obtain the digital clone's reply text;

[0026] The response text is input into the trained speech synthesis model to synthesize the response audio and the Mel spectrum of the audio. The synthesized response audio is placed into an audio queue. Audio frames with the same length as the driving video frame are obtained according to the video frame rate. The Mel spectrum features of each audio frame are placed into the Mel spectrum queue in sequence. An audio tag sequence is established with a sequence length equal to the number of audio frames. The first tag has a true value, which indicates that the time corresponding to the tag is the start time of the audio. The values ​​of the remaining tags are false. The audio tags in the sequence are placed into the audio tag queue in sequence.

[0027] The answer text is segmented using an existing word segmentation model. Based on the word segmentation results, body movements are inserted to obtain a sequence of body movement video frames. The sequence of body movement video frames is then placed into a queue of body movement video frames.

[0028] 2) If the voice interaction information or interaction action does not exist, it means that the digital clone does not need to generate a response or body action. The driving video frame with the smallest sequence number that has not been used for the longest time is selected and placed into the body action video frame queue. A false flag is placed in the audio flag queue and an empty feature is placed in the Mel spectrum queue to indicate that there is no audio content in this time period.

[0029] The method described above is characterized by the following specific process in the lip-sync step during the digital clone generation stage:

[0030] Extract the Mel spectrum features from the head of the Mel spectrum queue, extract the limb movement video frame from the head of the limb movement video frame queue, and evaluate the Mel spectrum features:

[0031] 1) If the Mel spectrum feature is not empty, input the Mel spectrum feature and the limb movement video frame into the lip generation model to synthesize a lip-synced video frame that is synchronized with the audio frame corresponding to the Mel spectrum feature, and place the lip-synced video frame into the queue to be enhanced.

[0032] 2) If the Mel spectrum feature is empty, it means that lip-sync is not required, and the video frame of the limb movement is directly placed into the queue to be enhanced.

[0033] The method described above is characterized by the following specific process in the facial enhancement step of the digital clone generation stage:

[0034] Take the video frame to be enhanced from the head of the queue, input the video frame to be enhanced into the face enhancement model to obtain an enhanced video frame with higher facial clarity, and put the enhanced video frame into the output queue.

[0035] The audio and video output includes:

[0036] Retrieve the enhanced video frame from the head of the output queue, display the enhanced video frame, and determine the display duration based on the video frame rate. Retrieve the audio tag from the head of the audio tag queue and perform a judgment:

[0037] If the audio flag value is true, the audio at the head of the audio queue is retrieved and played asynchronously to achieve audio-visual synchronization.

[0038] If the audio tag value is false, no further action will be taken.

[0039] The method described above is characterized in that, in the perception and response step, a text response is generated based on voice interaction information or interactive actions to obtain the response text of the digital clone, including:

[0040] If there is voice interaction information, the existing speech recognition model is used to convert the speech into text in real time, and the converted text is input into the existing question-and-answer language model to obtain the answer text.

[0041] If only interactive actions exist, the existing action recognition model is used to identify the action, and the action-response relationship is used for matching to obtain the response text.

[0042] The method described above is characterized in that, in the perception and response step, body movement insertion is performed based on the word segmentation results to obtain a body movement video frame sequence, including:

[0043] The longest unused video frame with the smallest sequence number is selected as the driving video frame to be used, and the trigger keyword is used to match the word segmentation result.

[0044] If a match is successful, the existing frame interpolation model is used to perform frame interpolation, with the last frame of the limb action video frame queue and the first frame of the action video corresponding to the matched keyword as the start frame and the end frame, respectively. The last frame of the action video and the driving video frame to be used are used as the start frame and the end frame, respectively, to achieve a smooth transition of discontinuous image content between the limb action video frames already in the limb action video frame queue, the action video frames corresponding to the matched keyword, and the driving video frames to be used. The frames generated by the frame interpolation model, the action video frames, and the obtained driving video frames are combined in chronological order to form a limb action video frame sequence.

[0045] If no keyword match is found, the driving video frame to be used will be used as a sequence of limb movement video frames with a length of 1.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. Possesses both visual and auditory sensory abilities, enabling it to recognize real people's words and gestures and respond naturally;

[0048] 2. Digital avatars offer a variety of expressive channels, including voice, facial expressions, and gestures;

[0049] 3. It consumes few computing resources and has low response latency, enabling deployment on consumer-grade home PCs and meeting real-time dialogue requirements;

[0050] 4. It can simultaneously handle the dialogue needs of both open-ended dialogue scenarios and specialized business scenarios. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the real-time dialogue-based digital avatar generation method with audiovisual perception capabilities proposed in this invention.

[0052] Figure 2 This is a schematic diagram of the process for inserting limb movements as proposed in this invention;

[0053] Figure 3 This is a schematic diagram of the lip-sync process proposed in this invention;

[0054] Figure 4 This is a schematic diagram of the facial enhancement process proposed in this invention. Detailed Implementation

[0055] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present application, and should not be construed as limiting the present application.

[0056] The method for generating a digital clone according to an embodiment of this application is described below with reference to the accompanying drawings.

[0057] Example 1

[0058] Figure 1 A flowchart illustrating a real-time conversational digital avatar generation method with audiovisual perception capabilities, provided in an embodiment of the present invention, is shown in the figure, including:

[0059] During the training and preprocessing phases, audio and video materials are recorded and preprocessed, and then used to train the speech synthesis model, lip-shape generation model, and face enhancement model.

[0060] A buffer is established during the digital avatar generation stage. Through four steps—perception and response, lip-syncing, facial enhancement, and audio-visual output—a digital avatar for real-time dialogue is generated.

[0061] In this embodiment, the training and preprocessing phase involves recording and preprocessing audio and video materials, including:

[0062] Record audio and video materials of a real-life prototype digital avatar under the same hardware conditions and environment, including voice audio, driving video, motion video, and training audio and video. The voice audio is an audio clip of the real-life prototype reading a pre-prepared text. The driving video is a video clip of the real-life prototype in a natural state without large facial movements and with both hands in a designated position on the body. The motion video consists of multiple videos of the real-life prototype performing various hand gestures. The hand gestures will be used as preset actions when inserting actions. At the beginning and end of each action, the person's hands must remain in the designated position where the hands were held in the driving video to facilitate the transition with the driving video. The training audio and video consists of one or more audio and video clips of the real-life prototype reading aloud in a natural state.

[0063] Preprocessing includes:

[0064] Specify the video frame rate, preferably 25 FPS, and adjust all video footage to this frame rate;

[0065] Using the existing frame interpolation model, the last frame of the driving video is used as the starting frame and the first frame of the driving video is used as the ending frame for frame interpolation. A set of composite frames synthesized by the frame interpolation model is obtained, which can make the beginning and end of the driving video transition naturally. These composite frames are then placed at the end of the driving video in chronological order. Specifically, assuming that the last frame of the driving video is the Nth frame, the k composite frames obtained by interpolating between the starting frame and the ending frame are regarded as the N+1 to N+k frames in sequence.

[0066] Define a set of trigger keywords for body movements, and determine the corresponding action videos for each keyword;

[0067] Define a set of action-response relationships to determine the text of the fixed response voice corresponding to various physical action interaction inputs received by the digital avatar system from the user.

[0068] In this embodiment, training the speech synthesis model, lip-shape generation model, and face enhancement model using audio and video materials includes:

[0069] The existing speech synthesis model, pre-trained on a publicly available large speech dataset, is fine-tuned using the speech audio and the text corpus to obtain the speech synthesis model. The input of the speech synthesis model is text, and the output is the audio corresponding to the text and the Mel spectrum of the audio. The preferred speech synthesis model is the FastSpeech2 model.

[0070] The audio of the training audio-visual video is extracted and converted into Mel spectrogram features with the same frame rate as the video. The existing audio-driven lip-sync model pre-trained with a large amount of publicly available audio-visual data is fine-tuned using the RGB frames of the training audio-visual video and the Mel spectrogram features of the audio in the training audio-visual video to obtain the lip-sync generation model. The audio-driven lip-sync model is preferably the Wav2Lip model.

[0071] The Mel spectrum of the audio in the training audio-video and the driving video are input into the lip-shape generation model to obtain a low-resolution synthetic video synchronized with the audio in the training audio-video. The synthetic video and the video in the training audio-video are used as source data and target data, respectively, to train an existing face replacement network to obtain the face enhancement model. The existing face replacement model is preferably the SAEHD model of DeepFaceLab.

[0072] In this embodiment, establishing a buffer during the digital clone generation stage includes:

[0073] Extract all RGB frames of the driving video and number them in ascending order of time;

[0074] A buffer is established with a limb movement video frame queue, a queue to be enhanced, an audio queue, a Mel spectrum queue, an audio tag queue, and an output queue. The limb movement video frame queue stores limb movement video frames, the queue to be enhanced stores video frames to be enhanced, the audio queue stores audio to be played, the Mel spectrum queue stores the Mel spectrum features of the audio to be played, the audio tag queue stores Boolean tags indicating whether audio should be played, and the output queue stores video frames that have been enhanced for the final output.

[0075] In this embodiment, the perception and response include:

[0076] The system monitors and acquires real-time audio data, while the camera captures video data from the user. It determines the presence of voice interaction information in the audio data based on audio loudness, and identifies the presence and type of interactive actions in the video data based on existing human keypoint recognition models.

[0077] 1) If the aforementioned voice interaction information or interaction action exists:

[0078] Based on voice interaction information or interactive actions, text responses are generated to obtain the digital clone's reply text, specifically:

[0079] If there is voice interaction information, the existing speech recognition model is used to convert the speech into text in real time, and the converted text is input into the existing question-and-answer language model to obtain the answer text.

[0080] If only interactive actions exist, the existing action recognition model is used to identify the action, and the action-response relationship is used for matching to obtain the response text.

[0081] In this embodiment, obtaining the digital clone's response text by providing text replies based on voice interaction information or interactive actions includes:

[0082] If there is voice interaction information, the existing speech recognition model is used to convert the speech into text in real time, and the converted text is input into an existing question-and-answer language model provided by a third party to obtain the answer text.

[0083] If only interactive actions exist, the existing action recognition model is used to identify the action, and the action-response relationship is used for matching to obtain the response text.

[0084] The answer text is input into the trained speech synthesis model to synthesize the answer audio and the Mel spectrum of the audio. The synthesized answer audio is placed into the audio queue. Taking the video frame rate of 25 FPS as an example, the video is divided into audio frames with a frame length of 40 ms according to 25 FPS. The Mel spectrum features of each audio frame are placed into the Mel spectrum queue in sequence to establish an audio tag sequence. The sequence length is the number of audio frames. The first tag has a value of true, which is used to indicate that the time corresponding to the tag is the start time of the audio. The values ​​of the remaining tags are false. The audio tags in the sequence are placed into the audio tag queue in sequence.

[0085] The response text is segmented using an existing word segmentation model, preferably jieba. Based on the segmentation results, body language actions are inserted to obtain a sequence of body language action video frames. This sequence is then sequentially placed into a body language action video frame queue. Specifically:

[0086] The longest unused video frame with the smallest sequence number is selected as the driving video frame to be used, and the trigger keyword is used to match the word segmentation result.

[0087] If the match is successful, then as follows Figure 2 As shown, using the existing frame interpolation model, the last frame of the limb action video frame queue and the first frame of the action video corresponding to the matched keyword are used as the start frame and the end frame, respectively, for frame interpolation. The last frame of the action video and the driving video frame to be used are used as the start frame and the end frame for frame interpolation. This achieves a smooth transition of discontinuous image content between the limb action video frames already in the limb action video frame queue, the action video frames corresponding to the matched keywords, and the driving video frames to be used. The frames generated by the frame interpolation model, the action video frames, and the obtained driving video frames are combined in chronological order to form a limb action video frame sequence.

[0088] If no keyword match is found, the driving video frame to be used will be used as a sequence of limb movement video frames with a length of 1.

[0089] 2) If the voice interaction information or interaction action does not exist, it means that the digital clone does not need to generate a response or body action. The driving video frame with the smallest sequence number that has not been used for the longest time is selected and placed into the body action video frame queue. A false flag is placed in the audio flag queue and an empty feature is placed in the Mel spectrum queue to indicate that there is no audio content in this time period.

[0090] In this embodiment, the lip-sync includes:

[0091] Extract the Mel spectrum features from the head of the Mel spectrum queue, extract the limb movement video frame from the head of the limb movement video frame queue, and evaluate the Mel spectrum features:

[0092] 1) If the Mel spectrum feature is not empty, then as follows Figure 3 As shown, the Mel spectrum features and the limb movement video frame are input into the lip-shape generation model to synthesize a lip-shape synchronized video frame that is synchronized with the audio frame corresponding to the Mel spectrum features, and the lip-shape synchronized video frame is placed into the queue to be enhanced.

[0093] 2) If the Mel spectrum feature is empty, it means that lip-sync is not required, and the video frame of the limb movement is directly placed into the queue to be enhanced.

[0094] In this embodiment, the facial enhancement includes:

[0095] like Figure 4 As shown, the video frame to be enhanced at the head of the queue is taken out from the queue to be enhanced, the facial region is cropped, the cropped facial region is input into the facial enhancement model to obtain a higher resolution facial image, the obtained facial image and the video frame to be enhanced are stitched together to obtain an enhanced video frame with higher facial resolution, and the enhanced video frame is placed into the output queue.

[0096] In this embodiment, the audio and video output includes:

[0097] Retrieve the enhanced video frame from the head of the output queue, display the enhanced video frame, and determine the display duration based on the video frame rate. Retrieve the audio tag from the head of the audio tag queue and perform a judgment:

[0098] If the audio flag value is true, the audio at the head of the audio queue is retrieved and played asynchronously to achieve audio-visual synchronization.

[0099] If the audio tag value is false, no further action will be taken.

Claims

1. A real-time conversational digital avatar generation method with audio-visual perceptual capabilities, characterized by, The steps are as follows: In the training and preprocessing stage, audio and video materials are recorded and preprocessed, and audio synthesis model, lip generation model and face enhancement model are trained using audio and video materials; In the digital avatar generation stage, a buffer is established, and a queue of body action video frames, a queue to be enhanced, an audio queue, a mel spectrum queue, an audio marker queue and an output queue are obtained; Real-time audio data is obtained by listening, and video data of the user is collected by a camera. Whether there is voice interaction information in the audio data is determined according to the audio loudness, and whether there is an interactive action in the video data is determined according to the existing human key point recognition model, and the type of the interactive action is recognized: 1) If the voice interaction information or the interactive action exists, a text reply is performed according to the voice interaction information or the interactive action, and a reply text of the digital avatar is obtained; The reply text is input into the trained voice synthesis model to synthesize reply audio and the mel spectrum of the audio. The synthesized reply audio is placed in the audio queue, and the audio frames with the same length as the driving video frames are obtained by cutting according to the video frame rate. The mel spectrum features of each audio frame are sequentially placed in the mel spectrum queue, an audio marker sequence is established, the sequence length is the number of audio frames, the value of the first marker is true, indicating that the time corresponding to the marker is the starting time of the audio, and the values of the remaining markers are false. The audio markers in the sequence are sequentially placed in the audio marker queue. The reply text is segmented using an existing segmentation model, and body action insertion is performed according to the segmentation result to obtain a sequence of body action video frames, which are sequentially placed in the body action video frame queue; 2) If the voice interaction information or the interactive action does not exist, it means that the digital avatar does not need to generate a reply or a body action. The longest time unused and the smallest sequence number of a frame of driving video frame is taken, and the driving video frame is placed in the body action video frame queue. A marker with a value of false is placed in the audio marker queue, and a feature with a value of empty is placed in the mel spectrum queue to indicate that there is no audio content in this time period; The mel spectrum feature at the head of the queue is taken from the mel spectrum queue, and the body action video frame at the head of the queue is taken from the body action video frame queue. The mel spectrum feature is judged: 1) If the mel spectrum feature is not empty, the mel spectrum feature and the body action video frame are input into the lip generation model to synthesize a lip synchronization video frame synchronized with the audio frame corresponding to the mel spectrum feature, and the lip synchronization video frame is placed in the to-be-enhanced queue; 2) If the mel spectrum feature is empty, it means that lip synchronization is not needed, and the body action video frame is directly placed in the to-be-enhanced queue; After face enhancement and audio and video output, a digital avatar for real-time conversation is generated.

2. The real-time conversational digital avatar generation method with audio-visual perceptual capability according to claim 1, wherein, In the training and preprocessing stage, audio and video materials are recorded and preprocessed, including: recording audio and video materials of a real prototype of a digital avatar, including speech audio, driving video, action video and training audio and video, wherein the speech audio is an audio content of the real prototype reading a pre-prepared text corpus, the driving video is a video of the real prototype in a natural state without large facial movements and with both hands kept at a specified position of the body, the action video is a plurality of videos of the real prototype performing various gestures, wherein the gestures will be used as preset actions when inserted, at the beginning and end of each action, the hands of the prototype are kept at the specified position of the driving video to facilitate the transition with the driving video, and the training audio and video is one or more audio and videos of the real prototype reading in a natural state; the preprocessing includes: specifying a video frame rate and adjusting all video materials to the frame rate; using an existing frame insertion model to insert frames from the last frame of the driving video as the starting frame and the first frame of the driving video as the ending frame, to obtain a set of synthesized frames that can make the driving video transition naturally at the beginning and end, and place them in the tail of the driving video in chronological order, specifically, assuming the last frame of the driving video is the Nth frame, the k synthesized frames obtained by inserting frames between the starting frame and the ending frame are sequentially regarded as the N+1th to N+kth frames; defining a set of trigger keywords for limb actions and determining the action video corresponding to each keyword; defining a set of action-answer relationships and determining the text of the fixed answer voice corresponding to various limb action interactions received by the digital avatar system.

3. The real-time conversational digital avatar generation method with audio-visual perceptual capability according to claim 1, wherein, training a speech synthesis model, a lip generation model and a face enhancement model using audio and video materials, including: fine-tuning an existing speech synthesis model pre-trained on a large public speech dataset using the speech audio and the text corpus to obtain the speech synthesis model, the input of which is text and the output of which is audio and the mel spectrum of the audio; extracting the audio of the training audio and video, converting the audio into mel spectrum features with the same frame rate as the video, and fine-tuning an existing audio-driven lip synchronization model pre-trained on a large amount of public audio and video data using the RGB frames of the training audio and video and the mel spectrum features of the audio in the training audio and video to obtain the lip generation model; inputting the mel spectrum of the audio in the training audio and video and the driving video into the lip generation model to obtain a low-resolution synthesized video synchronized with the audio in the training audio and video, and using the synthesized video and the video in the training audio and video as source data and target data respectively to train an existing face replacement network, i.e., to obtain the face enhancement model.

4. The real-time conversational digital avatar generation method with audio-visual perceptual capability according to claim 1, wherein, establishing a buffer in the digital avatar generation stage, including: extracting all RGB frames of the driving video and labeling them with sequence numbers in chronological order from small to large; A limb action video frame queue, a to-be-enhanced queue, an audio queue, a mel spectrum queue, an audio mark queue, and an output queue are established as buffers, wherein the limb action video frame queue stores limb action video frames, the to-be-enhanced queue stores video frames to be subjected to face enhancement, the audio queue stores audio to be played, the mel spectrum queue stores mel spectrum features of the audio to be played, the audio mark queue is used to store a Boolean mark indicating whether the audio should be started to be played, and the output queue stores video frames subjected to face enhancement and used for final output.

5. The real-time conversational digital avatar generation method with audio-visual perceptual capability of claim 1, wherein, The face enhancement comprises: A to-be-enhanced video frame at a queue head is taken out from the to-be-enhanced queue, the to-be-enhanced video frame is input into the face enhancement model, an enhanced video frame with higher face definition is obtained, and the enhanced video frame is placed into the output queue.

6. The real-time conversational digital avatar generation method with audiovisual perceptual capabilities of claim 1, wherein, The audio-video output comprises: An enhanced video frame at a queue head is taken out from the output queue, the enhanced video frame is displayed, a display duration is determined according to the video frame rate, a head audio mark is taken out from the audio mark queue, and a judgment is made: If the audio mark value is true, audio at a queue head is taken out from the audio queue, the audio is started to be played asynchronously, and audio-visual synchronization is realized; If the audio mark value is false, no additional operation is performed.

7. The real-time conversational digital avatar generation method with audiovisual perceptual capabilities of claim 1, wherein, Text reply is made according to voice interaction information or interactive action, and an answer text of the digital avatar is obtained, comprising: If there is voice interaction information, the voice is converted into text in real time using an existing voice recognition model, the converted text is input into an existing question-answer language model, and the answer text is obtained; If there is only interactive action, the action is recognized using an existing action recognition model, and the answer text is obtained by matching according to an action-answer relationship.

8. The real-time conversational digital avatar generation method with audiovisual perceptual capabilities of claim 1, wherein, Limb action insertion is made according to a word segmentation result, and a limb action video frame sequence is obtained, comprising: A driving video frame with the longest time not used and the smallest sequence number is taken as a to-be-used driving video frame, and the word segmentation result is matched using a trigger keyword; If the matching is successful, an existing frame insertion model is used, the last frame of the limb action video frame queue and the first frame of an action video corresponding to the matched keyword are taken as a starting frame and a terminating frame respectively, the last frame of the action video and the to-be-used driving video frame are taken as the starting frame and the terminating frame respectively, the discontinuous image content between the limb action video frame already in the limb action video frame queue, the action video frame corresponding to the matched keyword, and the to-be-used driving video frame is smoothly transitioned, the frame generated by the frame insertion model, the action video frame, and the obtained driving video frame are combined into the limb action video frame sequence in time sequence; If no keyword matching is successful, the to-be-used driving video frame is taken as a limb action video frame sequence with a length of 1.

Images