Digital human real-time interaction system and digital human real-time interaction method

Abstract

The invention discloses a digital human real-time interaction system and method. The interaction system comprises terminal equipment and a cloud server. The terminal device receives voice input by a user and transmits the voice to the cloud server. The cloud server comprises a video acquisition and processing module, an intelligent question and answer and voice generation module, an audio processing module and an image rendering module. A video capture and processing module captures original video data indicative of an object of a digital figure and generates successive image frames. The intelligent question and answer and voice generation module converts the voice into a text; performing semantic analysis on the text, understanding the intention of the user, and generating a corresponding reply text; the reply text is also converted into a reply voice. The audio processing module processes the reply voice and the continuous image frames to generate first video data. And the image rendering module fuses the first video data with the virtual background, and performs real-time processing on the fused video to generate a response video. And the terminal equipment outputs the response video.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a digital human real-time interaction system and a digital human real-time interaction method. Background Art

[0002] With the rapid development of virtual reality (VR), augmented reality (AR) and artificial intelligence (AI), digital human technology has gradually become a key component of future human-computer interaction. Existing digital humans mainly rely on pre-recorded audio and video content and lack real-time personalized interaction. On the other hand, the virtual character's lip sync and voice are poorly synchronized and the performance is unnatural, resulting in a poor interactive experience. Traditional lip sync technology usually relies on a lot of manual adjustments or inefficient animation production processes, which cannot meet the needs of real-time interaction.

[0003] Therefore, a new digital human interaction solution is needed to overcome the above problems. Summary of the invention

[0004] The present application provides a digital human real-time interaction system and a digital human real-time interaction method, in an effort to solve or at least alleviate at least one of the above problems.

[0005] According to one aspect of the present application, a digital human real-time interaction system is provided, comprising: a terminal device, comprising an audio input module and a video output module, the terminal device being adapted to receive a user's voice input via the audio input module and transmit it to a cloud server, and also adapted to output a response video to the user's voice via the video output module; a cloud server being adapted to process the user's voice input in real time to generate the response video, and transmit the response video to the terminal device, the cloud server comprising: a video acquisition and processing module being adapted to acquire original video data of an object indicating a digital human image, wherein the original video data includes the object's body movements and facial expressions, and to generate a video containing only the object's body movements and facial expressions. The invention relates to a continuous image frame; an intelligent question-answering and speech generation module, which is suitable for converting the speech input by the user into text; performing semantic analysis on the text, understanding the user's intention, and generating corresponding reply text using a language model; and also suitable for converting the generated reply text into reply speech; an audio processing module, which is suitable for processing the reply speech and the continuous image frames based on an audio-visual synchronization model to generate an audio-driven image frame as the first video data, wherein the lip shape of the first video data is synchronized with the reply speech; an image rendering module, which is suitable for fusing the first video data with a virtual background, and performing real-time processing on the fused video frames to generate the response video, wherein the virtual background includes static images, dynamic videos, and three-dimensional scenes.

[0006] Optionally, in the system according to the present application, in the cloud server, the video acquisition and processing module includes: a video acquisition unit, suitable for acquiring original video data using a green screen background, and capturing the body movements and facial expressions of the object to obtain original video data; a video processing unit, suitable for processing the original video data to obtain the continuous image frames, including: using an algorithm based on chromaticity difference to remove the green screen background in the original video data to obtain transparent channel video data containing the object, sharpening the edges in the transparent channel video data to obtain processed transparent channel video data, and correcting the processed transparent channel video data to generate continuous image frames with consistent timing.

[0007] Optionally, in the system according to the present application, in the cloud server, the audio processing module is also suitable for training and generating an audio-visual synchronization model based on a neural network, and the audio-visual synchronization model is suitable for extracting features of the input reply voice, generating spectral features corresponding to the reply voice, analyzing the input continuous image frames, extracting visual features of the object in different facial expressions and lip states, and matching the spectral features and the visual features to generate a video with synchronized voice and lip shape.

[0008] Optionally, in the system according to the present application, in the cloud server, the audio processing module includes: a text conversion unit, suitable for converting natural language text into audio data; a training unit, suitable for generating training samples using the audio data and its corresponding real video data, and using a pre-trained model to train and generate an audio-visual synchronization model through the training samples, wherein the pre-trained model is a deep learning model based on Wav2Lip.

[0009] Optionally, in the system according to the present application, in the cloud server, the intelligent question and answer and speech generation module includes: an automatic speech recognition unit, suitable for using an acoustic model based on a deep neural network to recognize the speech input by the user and obtain text corresponding to the speech; a natural language processing unit, suitable for using a language model to deeply understand the text, determine the user's intention, and generate a reply text for the user's input speech based on the user's intention; a text-to-speech unit, suitable for converting the reply text into a reply speech through a neural network speech synthesis method.

[0010] Optionally, in the system according to the present application, in the cloud server, the natural language processing unit includes: a text preprocessing subunit, suitable for processing the text using a word segmentation algorithm to obtain multiple word segments; an intention recognition subunit, suitable for performing semantic analysis and context understanding on multiple word segments of the current text and the dialogue history through a dialogue state tracking algorithm to determine the current dialogue state, wherein the dialogue state includes the user's intention; a dialogue management subunit, suitable for obtaining required information from a relevant knowledge base, generating an initial reply text, and performing semantic and grammatical checks on the initial reply text to obtain the reply text.

[0011] Optionally, in the system according to the present application, in the cloud server, the intelligent question and answer and speech generation module is also suitable for accessing an external knowledge base through an interface.

[0012] Optionally, in the system according to the present application, in the cloud server, the image rendering module is also suitable for preprocessing the virtual background according to the actual application scenario, if the virtual background is a static background, the preprocessing includes adjusting the resolution, color space and frame rate; if the virtual background is a dynamic background, the preprocessing includes using a time series-based synchronous rendering technology to ensure the consistency of the movement of the virtual background and the object; the preprocessed virtual background is merged with the first video data, including: dynamically adjusting the position and intensity of the light source in the preprocessed virtual background through a physically based rendering technology, and when the preprocessed virtual background is a three-dimensional scene, using a depth map-based rendering technology to add a sense of hierarchy to the object and the virtual background to achieve a realistic representation of spatial depth; using a real-time rendering engine to process the fused video images in real time, and output a high-resolution response video.

[0013] Optionally, in the system according to the present application, the image rendering module is also suitable for automatically adjusting the virtual background according to ambient lighting, user actions and voice commands.

[0014] Optionally, in the system according to the present application, the terminal device further includes: an audio signal processing module, adapted to perform preliminary processing on the voice input by the user.

[0015] According to another aspect of the present application, a method for real-time interaction of a digital human is provided, which is suitable for execution in a cloud server, and the method comprises: converting a voice input by a user into text; performing semantic analysis on the text to determine the user's intention, and generating a corresponding reply text based on the user's intention; converting the generated reply text into a reply voice; based on an audio-visual synchronization model, processing the reply voice and pre-generated continuous image frames to generate audio-driven image frames as first video data, wherein the lip shape of the first video data is synchronized with the reply voice, and the continuous image frames contain a digital human image; fusing the first video data with a virtual background, and processing the fused video frames in real time to generate a response video to the user's voice.

[0016] Optionally, the method according to the present application also includes the step of pre-generating continuous image frames: collecting original video data of an object indicating a digital human image, wherein the original video data includes body movements and facial expressions of the object; processing the original video data to obtain continuous image frames containing only the object.

[0017] According to another aspect of the present application, a computing device is provided, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method as described above.

[0018] According to another aspect of the present application, a computer-readable storage medium storing one or more programs is provided. The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method described above.

[0019] According to another aspect of the present application, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction implements the above-mentioned method when executed by a processor.

[0020] In summary, according to the solution of this application, many problems faced by digital humans in the prior art when interacting with users, such as unnatural lip synchronization and poor real-time performance, have been solved. This system has high practicality and can bring users a more natural and smooth interactive experience in practical applications. At the same time, it is also scalable and can be widely used in a variety of application scenarios, such as online education, virtual customer service, game entertainment and other fields, which has effectively promoted the deep integration of the virtual world and the real world, and opened up a new development direction for the future field of digital interaction.

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles applied herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. By reading the following detailed description in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. Throughout the application, the same reference numerals generally refer to the same parts or elements.

[0023] Figure 1 A schematic diagram of a scene of a digital human real-time interaction system 100 according to some embodiments of the present application is shown;

[0024] Figure 2 A schematic diagram of the structure of a digital human real-time interaction system 100 according to some embodiments of the present application is shown;

[0025] Figure 3 A schematic diagram of training an audio-visual synchronization model according to some embodiments of the present application is shown;

[0026] Figure 4 A schematic diagram of a process of a digital human real-time interaction method 400 according to some embodiments of the present application is shown;

[0027] Figure 5 A schematic diagram of a computing device 500 according to some embodiments of the present application is shown. DETAILED DESCRIPTION

[0028] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0029] In order to solve the technical bottlenecks in related technologies, this application proposes a real-time digital human interaction system, which dynamically generates real-time lip shape changes of virtual humans through a deep learning model, and combines it with an intelligent question-answering engine to achieve efficient voice and lip-sync interaction. The system can be applied to real-time interactive scenarios such as online education, virtual customer service, and commercial exhibitions, providing a smooth and natural human-computer interaction experience.

[0030] Figure 1 A schematic diagram of a scene of a digital human real-time interaction system 100 according to some embodiments of the present application is shown.

[0031] like Figure 1 As shown, the digital human real-time interaction system 100 includes one or more terminal devices 110 and one or more cloud servers 120. The user can interact with the digital human through voice. When the user needs to give instructions or ask questions, he inputs the corresponding voice on the terminal device 110, such as "What will the weather be like in the next week?", "Please play an ancient poem related to wind", etc. The terminal device 110 transmits the voice input by the user to the cloud server 120.

[0032] The cloud server 120 is responsible for real-time processing of the user's input voice and generating a reply voice through the intelligent question-answering engine. Afterwards, the cloud server 120 generates a lip-driven video based on the reply voice, i.e., the first video data. In the first video data, the digital human has natural and smooth changes in character expression, and the lip shape and expression changes are highly consistent with the reply voice. Afterwards, the cloud server 120 merges the first video data with the virtual background to generate a response video for the user's voice.

[0033] Finally, the response video is synchronously played by the terminal device 110. The improvement of user experience is reflected in the natural and smooth dialogue and the dynamic performance of the digital human, which enhances the sense of participation and interactivity.

[0034] According to the implementation mode of the present application, the cloud server 120, as the core processing component of the system 100, is equipped with powerful computing resources, including high-performance multi-core processors and graphics processing units (GPUs), supporting parallel computing to meet the needs of large-scale users online at the same time. The voice signal is first transcribed by automatic speech recognition (ASR) to convert the voice into text information. Subsequently, natural language processing (NLP) is used for semantic understanding and analysis. NLP uses deep learning algorithms and language models trained based on large-scale corpora to identify user intentions and generate intelligent responses.

[0035] In some embodiments, the intelligent question-answering engine is an important component of the cloud server 120. Based on the knowledge graph and machine learning technology, it can quickly retrieve relevant information and generate natural and fluent text answers. The engine supports context understanding, can handle continuous questions from users, and adjust the reply content according to the interaction history to improve user satisfaction.

[0036] In some other embodiments, the cloud server 120 can also be connected to other servers. Specifically, the intelligent question-answering engine can also access other servers (such as external knowledge bases) through an API interface to achieve dynamic update and expansion of information and enhance the adaptability and flexibility of the system 100.

[0037] The configuration of the terminal device 110 is flexible and diverse, supporting various forms such as personal computers, mobile devices, and smart terminals. The terminal device 110 has a built-in audio input module (such as a microphone) and a video output module (such as a display screen) to ensure that the user can interact with the digital human conveniently.

[0038] In some other embodiments, the terminal device 110 integrates basic processing capabilities and can perform preliminary analysis of the voice signal input by the user through edge computing to reduce the computing power burden of the cloud server 120 and improve the overall response speed of the system 100.

[0039] like Figure 1, the system 100 also includes a network 130. The terminal device 110 is connected to the cloud server 120 through an efficient network communication architecture. The system 100 uses the Real-Time Transport Protocol (RTP) and the User Datagram Protocol (UDP) to reduce network latency and achieve rapid transmission of audio and video data. The interaction between the terminal device 110 and the cloud server 120 is achieved through WebSocket to achieve two-way real-time communication, ensuring the immediacy of audio input and system response. In addition, in order to improve the stability and security of the system 100, an encrypted transmission protocol is used to protect the privacy and security of user data.

[0040] According to some other embodiments of the present application, the terminal device 110 can integrate basic processing capabilities. For example, the terminal device 110 also includes an audio signal processing module (not shown). The audio signal processing module is coupled to the audio input module 112 to perform preliminary processing on the voice input by the user. The preliminary processing includes noise removal, signal enhancement, etc., which is intended to improve the audio quality and provide a better basis for subsequent processing and analysis. The terminal device 110 sends the preliminarily processed voice to the cloud server 120 for processing to reduce the burden on the cloud server 120 and improve the overall response speed of the system.

[0041] The digital human real-time interaction system 100 has high flexibility and scalability, and can be widely deployed in various application scenarios, including social, entertainment, education, marketing and other fields.

[0042] In the social field, the system 100 can be integrated with an instant messaging platform to support users to interact with virtual digital people through voice or text. Combined with a multi-language support module, the system 100 can process inputs in multiple languages ​​and dialects, providing customized interactive experiences for users around the world. At the same time, the emotional analysis function in the system 100 can identify the emotional state in the user's voice, adjust the expression and tone of the virtual digital person, and enhance the user's emotional interactive experience.

[0043] In the field of entertainment, digital humans can be used as virtual hosts, actors, or interactive characters in games, using efficient lip sync technology and smooth facial expression rendering to generate realistic virtual images in real time. In live broadcasts and video content creation, system 100 can support real-time generation of virtual anchors to enhance the audience's visual and interactive experience. In addition, the real-time animation rendering engine in system 100 can be seamlessly connected with the game engine to provide technical support for immersive entertainment content.

[0044] In educational scenarios, the system 100 can be used to create virtual teachers or trainers. By integrating speech recognition and natural language processing technologies, the system 100 can analyze students' questions in real time and provide intelligent feedback to support personalized learning. Combined with knowledge graphs and recommendation algorithms, the system 100 can generate customized teaching content to meet the needs of different students.

[0045] In the marketing field, virtual digital people can act as brand spokespersons or customer service assistants to interact with customers in real time. Through natural language processing technology, system 100 can understand user needs and provide personalized product recommendations or answer questions based on semantic analysis. The expanded multimedia fusion function allows virtual characters to be embedded in dynamic advertisements or interactive videos to enhance user engagement and brand interaction effects.

[0046] This multi-scenario applicability enables the system 100 to flexibly adapt to various complex interaction needs, and has strong scalability, and can be customized and optimized according to the specific needs of different fields.

[0047] Figure 2 A structural schematic diagram of a digital human real-time interaction system 100 according to some embodiments of the present application is shown.

[0048] like Figure 2 As shown, in addition to the desired configuration, the terminal device 110 includes an audio input module 112 and a video output module 114. The terminal device 110 receives the voice input by the user via the audio input module 112 and transmits it to the cloud server 120. In addition, the terminal device 110 also outputs a response video to the user's voice via the video output module 114.

[0049] The cloud server 120 processes the voice input by the user in real time to generate a response video, and transmits the response video to the terminal device 110 .

[0050] In some embodiments, the cloud server 120 includes: a video acquisition and processing module 122 , an intelligent question and answer and voice generation module 124 , an audio processing module 126 and an image rendering module 128 .

[0051] The video acquisition and processing module 122 acquires raw video data of an object indicating a digital human image, wherein the raw video data includes the object's body movements and facial expressions, and generates continuous image frames containing only the object. The object can be a person, an animal, a virtual image, etc., and this application does not limit this.

[0052] The intelligent question-answering and voice generation module 124 obtains the voice input by the user from the terminal device 110, and generates a corresponding reply voice based on the voice. Specifically, the voice input by the user is first converted into text, and then the text is semantically analyzed to understand the user's intention, and the corresponding reply text is generated using the language model; then, the generated reply text is converted into a reply voice.

[0053] The audio processing module 126 is coupled to the video acquisition and processing module 122 and the intelligent question and answer and speech generation module 124 respectively. In this way, based on the audio-visual synchronization model, the audio processing module 126 processes the reply speech from the intelligent question and answer and speech generation module 124 and the continuous image frames from the video acquisition and processing module 122 to generate audio-driven image frames as the first video data. According to this embodiment, in the first video data, the lip shape of the object is synchronized with the reply speech.

[0054] The image rendering module 128 is coupled to the audio processing module 126, and uses a multi-channel video mixing technology to merge the first video data with the virtual background, and processes the merged video frames in real time to generate the response video. According to this embodiment, the virtual background includes static images, dynamic videos, three-dimensional scenes, etc., and can be adjusted according to the application scenario and the content of the conversation, which is not limited in this application.

[0055] The following is a detailed description of each component in the cloud server 120.

[0056] The video acquisition and processing module 122 is responsible for the acquisition and preprocessing of the initial video material of the digital human image, and includes a coupled video acquisition unit 1222 and a video processing unit 1224 .

[0057] The video acquisition unit 1222 acquires the original video data using the green screen background, and captures the body movements and facial expressions of the object to obtain the original video data. Specifically, a professional green screen shooting technology is used, and a high-resolution camera is used to capture the facial expressions, body movements, etc. of the object, and transmits them to the video acquisition unit to obtain high-quality original video data. During the video acquisition process, it is necessary to ensure that the light is uniform to avoid shadows interfering with the color key processing of the background. In this embodiment, the acquired object represents a digital human image, and the object can be a person, an animal, etc.

[0058] The video processing unit 1224 processes the original video data to obtain continuous image frames. Specifically, first, the green screen background in the original video data is removed using an algorithm based on chromaticity difference to obtain transparent channel video data containing the object. At the same time, the edges in the transparent channel video data are sharpened (Edge Refinement) to eliminate the common aliasing and edge bleeding problems in the matting process, and the processed transparent channel video data is obtained. Afterwards, the processed transparent channel video data is corrected to generate continuous image frames with consistent timing to ensure the coherence and stability of the object's body movements and facial expressions in dynamic scenes. When performing timing correction on video data, a network model based on deep learning, such as STTN (Spatial-TemporalTransformations Network), can be used to fill in the missing content of all input image frames by inputting adjacent image frames and distant image frames to improve spatiotemporal consistency. This application does not limit this.

[0059] The intelligent question-answering and speech generation module 124 (i.e., the intelligent question-answering engine described above) integrates multiple artificial intelligence technologies, including automatic speech recognition (ASR), natural language processing (NLP), and text to speech (TTS). In some embodiments, the intelligent question-answering and speech generation module 124 includes: an automatic speech recognition unit 1242, a natural language processing unit 1244, and a text-to-speech unit 1246.

[0060] When the user inputs a voice, first, the automatic speech recognition unit 1242 uses an acoustic model based on a deep neural network to recognize the voice input by the user and obtain the text corresponding to the voice. In some embodiments, the automatic speech recognition unit 1242 uses an acoustic model based on a deep neural network (DNN) to achieve efficient transcription of the user's voice through feature extraction (such as Mel frequency cepstral coefficients MFCC) and classification of acoustic feature vectors. The model can cope with changes in different accents, speaking speeds, and environmental noise to ensure the accuracy of recognized text.

[0061] The recognized text is passed to the natural language processing unit 1244. The natural language processing unit 1244 uses the language model to deeply understand the text and determine the user's intention. Afterwards, the natural language processing unit 1244 generates a reply text for the user's input voice according to the user's intention. The language model is, for example, BERT, GPT, etc., but is not limited thereto. Optionally, a language model based on the Transformer architecture is used to process the text input by the user to obtain the corresponding reply text, which can quickly understand and generate high-quality conversation content, support complex language scenarios and multi-round conversations, and greatly improve the user experience.

[0062] Specifically, the natural language processing unit 1244 will strictly obtain the required information from the constructed knowledge base or through the Internet in real time and efficiently according to the specific context of the user's input voice and the keywords contained therein, and then generate a logically rigorous and reasonable text answer. In this processing process, multiple crucial steps such as text vectorization, entity recognition, and contextual reasoning are involved. These steps cooperate and work together to ensure that the reply text has a high degree of accuracy and close relevance, providing users with more natural and fluent response content, thereby improving the user experience and satisfaction.

[0063] In some embodiments, the natural language processing unit 1244 further includes: a text preprocessing subunit, an intent recognition subunit, and a dialogue management subunit (not shown in the figure).

[0064] First, the text preprocessing subunit processes the text using a word segmentation algorithm to obtain multiple word segments. Specifically, a word segmentation algorithm (such as Word2Vec or BERT model) is used to segment the text into multiple vocabulary units, and part-of-speech tagging is performed to remove meaningless stop words. At the same time, synonym replacement and spelling correction are performed to ensure the quality and accuracy of the final generated word vector.

[0065] Afterwards, the intent recognition subunit processes multiple word segments through a deep learning algorithm (such as a long short-term memory network LSTM or a transformer model Transformer) to extract the user's intent. According to some embodiments of the present application, context modeling technology is used to better understand the user's intent. Specifically, through the dialogue state tracking (DST) algorithm, semantic analysis and context understanding are performed on multiple word segments of the current text, as well as the dialogue history, to determine the current dialogue state, which includes user goals, user intentions, etc. Regarding the dialogue state tracking algorithm, existing models can be used to implement it, such as the NBT-DNN model, the NBT-CNN model, the end-to-end MemN2N model, etc., and the present application does not impose too many restrictions on this.

[0066] After identifying the user's intention, the dialogue management subunit generates a reply text based on the user's intention. Specifically, the dialogue management subunit obtains the required information from the relevant knowledge base (the relevant knowledge base includes a pre-built knowledge base and an external knowledge base accessed in real time through the network) to generate an initial reply text. Afterwards, the initial reply text is semantically and grammatically checked to ensure semantic rationality and grammatical correctness, and a logically rigorous and reasonable reply text is obtained. In addition, when generating a reply text, other multimodal information (such as expressions, intonation, etc.) can also be combined to enhance the user's immersion and interactive experience.

[0067] According to an embodiment of the present application, the intelligent question and answer and speech generation module 124 is also suitable for accessing an external knowledge base through an API interface, so that the natural language processing unit 1244 can retrieve information from the external knowledge base in real time when generating a reply text.

[0068] The generated reply text then enters the text-to-speech unit 1246, which converts the reply text into reply speech through a neural network speech synthesis method. In some embodiments, in addition to converting the reply text into speech using TTS, the text-to-speech unit 1246 also generates natural human voice feedback based on the intonation, rhythm and other information of the speech synthesis, combined with the speech content input by the user.

[0069] In addition, to further enhance the intelligence and interactive effect of the system 100, the natural language processing unit 1244 is provided with a real-time feedback mechanism. When interacting with the digital human, the user can let the system know his or her satisfaction through simple feedback (such as "yes" or "no"). Based on this feedback information, the system will continuously adjust and optimize the intent recognition and reply text generation algorithms to improve the accuracy and naturalness of subsequent conversations.

[0070] The intelligent question-answering and speech generation module 124 sends the generated reply speech to the audio processing module 126, together with the continuous image frames generated by the video acquisition and processing module 122, as the input of the audio-visual synchronization model. The audio-visual synchronization model is based on a neural network. On the one hand, it extracts features of the reply speech and generates spectral features corresponding to the reply speech. On the other hand, it analyzes the input continuous image frames and extracts the visual features of the object under different facial expressions and lip states. After that, it matches the spectral features and visual features to generate a video with synchronized speech and lip shape.

[0071] In some embodiments, the audio processing module 126 can also train and generate an audio-visual synchronization model. The audio-visual synchronization model uses the Wav2Lip model, which is specifically used for real-time matching of audio and lip shape. Its advantage is that it can handle unstable voice input and complex voice scenes, and can ensure the naturalness of voice and lip shape synchronization in various audio situations.

[0072] The audio processing module 126 also includes a text conversion unit 1262 and a training unit 1264 for training and generating an audio-visual synchronization model.

[0073] The text conversion unit 1262 converts the natural language text into audio data. In some embodiments, multiple natural language texts are collected and converted into audio data respectively.

[0074] In addition, for each natural language text, the character outputs the corresponding audio data (e.g., reciting, telling, etc.), and performs video capture to generate corresponding real video data. Usually, the real video data contains the character, especially the facial area of ​​the character, to reflect the lip movement of the person when speaking.

[0075] The training unit 1264 generates training samples using the audio data and its corresponding real video data, and uses a pre-trained model to train and generate an audio-visual synchronization model through the training samples. The pre-trained model is a deep learning model based on Wav2Lip.

[0076] Figure 3 FIG. 2 shows a schematic diagram of training an audio-visual synchronization model according to some embodiments of the present application. Figure 3 As shown, each real video data is processed. For example, each image frame in the real video data is partially occluded, and the area containing the lips is occluded as the occluded video data. Together with the audio data, it is input into the pre-trained model as a training sample. The model will adjust the lip movement of the character in real time according to the phoneme and timing information to generate a lip-synced video. The lip-synced video is compared with the real video data to adjust the model parameters. Through repeated iterative training, the model can automatically adapt to different speech rhythms and speaking speeds, so that the final generated video presents natural and smooth changes in character expressions, and the lip movement is highly consistent with the audio.

[0077] The image rendering module 128 uses image synthesis technology to merge the virtual background with the first video data, which specifically includes the following three steps.

[0078] The first step is to pre-process the virtual background according to the actual application scenario. If the virtual background is a static background (such as a static image), the pre-processing includes adjusting the resolution, color space (RGB or YUV), and frame rate to ensure synchronization and compatibility with the first video data. If the virtual background is a dynamic background, the pre-processing includes using a time series-based synchronous rendering technology to ensure the consistency of the virtual background and the object's movements.

[0079] The second step is to merge the preprocessed virtual background with the first video data. Specifically, during the synthesis process, the image rendering module 128 supports real-time calculation of dynamic light and shadow effects, and dynamically adjusts the position and intensity of the light source in the preprocessed virtual background through the physically based rendering (Physically Based Rendering, PBR) technology, so that the interaction between the object and the environment appears more realistic. In addition, if the preprocessed virtual background is a three-dimensional scene, the image rendering module 128 will also use depth map rendering technology (Depth Map Rendering) to add a sense of hierarchy to the object and virtual background, and realize the true expression of spatial depth.

[0080] The third step is to use the real-time rendering engine to process the fused video in real time after fusion and output high-resolution responsive video. The rendering engine supports hardware acceleration (GPU acceleration) and optimization based on multi-threaded processing to ensure that the final image is presented in a low-latency manner on high-performance devices.

[0081] According to the present application, the output response video can be played on various terminal devices 110 (such as smart large screens, desktops, mobile terminals, etc.), and users can achieve real-time interaction with digital humans through vision and voice.

[0082] In some other embodiments, the image rendering module 128 also supports automatic adjustment and switching of the virtual background according to the ambient lighting, user actions and voice commands, so as to achieve a more flexible and immersive interactive experience. For example, when the ambient lighting becomes dim, the virtual background can be switched to an image representing the night. For another example, according to the user's voice command, the virtual background that matches his intention can be switched.

[0083] In summary, according to the digital human real-time interaction system 100 of the present application, many problems faced by digital humans in the prior art when interacting with users, such as unnatural lip synchronization and poor real-time performance, are solved. This system has high practicality and can bring users a more natural and smooth interactive experience in practical applications. At the same time, it is also scalable and can be widely used in a variety of application scenarios, such as online education, virtual customer service, game entertainment and other fields, which has effectively promoted the deep integration of the virtual world and the real world, and opened up a new development direction for the future field of digital interaction.

[0084] Specifically, the system has the following advantages.

[0085] First, real-time

[0086] Relying on deep learning algorithms, combined with automatic speech recognition (ASR), natural language processing (NLP) and lip-driven technology, efficient and real-time interaction between digital humans and users is achieved. By adopting deep learning models, especially in the intelligent question-answering and speech generation modules, the advantages of pre-trained large language models (such as BERT, GPT, etc.) are fully utilized to quickly and accurately understand the user's voice input, and generate logically rigorous and accurate responses in a very short time through efficient dynamic reasoning mechanisms. This efficient data processing pipeline ensures that the intelligent feedback time after the user's voice input is stably maintained at the millisecond level, greatly improving the interaction efficiency, bringing users an extremely smooth and shocking interaction experience, and can effectively promote the development and progress of digital interaction technology.

[0087] In the audio processing module, Wav2Lip technology is used. This is a technology based on Generative Adversarial Network (GAN), which generates lip animations that are completely synchronized with the voice in real time through input audio. The advantage of Wav2Lip is that it is highly adaptable to different speaking speeds, accents, and voice emotions. It can adjust the lip shape changes of digital characters in real time to ensure that the lip shape and audio are highly matched, thereby enhancing the user's visual experience. In addition, the processing speed of the lip-driven is optimized by GPU parallelism and supports real-time streaming processing, ensuring that the character's expression and lip shape are still smooth and natural under high-load environments.

[0088] The system also uses an optimized low-latency rendering engine, GPU hardware acceleration and multi-threaded parallel processing technology to minimize the delay of audio and video synchronization rendering and achieve real-time high-resolution image output. In terms of multimedia integration, the system integrates real-time image synthesis and background rendering technology, which can seamlessly integrate digital character dynamics with virtual backgrounds, ensuring the stability and consistency of the image during real-time conversations.

[0089] This multi-module collaborative architecture design not only ensures the efficiency and real-time performance of the system, but also greatly improves the user's interactive experience, providing strong technical support for digital human interaction in various application scenarios.

[0090] Second, naturalness

[0091] In the digital human real-time interactive system, the Wav2Lip technology is used to ensure a high degree of synchronization between lip shape and audio, and to achieve a realistic and natural dynamic performance of lip shape and facial expressions. Compared with the traditional static model drive, the Wav2Lip model not only achieves high-precision lip shape matching, but can also dynamically adapt to changes in speech speed and intonation, so that the generated lip shape and audio show a more natural synchronization effect in any dialogue scene.

[0092] In the specific implementation, Wav2Lip technology deeply integrates the input speech features with the digital human facial expression model through joint modeling of audio features and visual features. This technology captures subtle changes in the audio signal and dynamically adjusts the shape of the mouth, the opening and closing amplitude, and the micro-expressions of the facial muscles, so that the mouth shape changes of the digital character are closer to the facial dynamics of humans when speaking naturally. At the same time, the system also has high robustness to different timbres, emotional states, accents, and speaking speeds. It can appropriately adjust facial expressions according to the emotional information in the audio, so that the character can show subtle facial changes in different emotional scenes, further enhancing the naturalness and affinity of the digital human.

[0093] In addition, the system combines facial motion capture technology to capture and analyze the changes in the digital human's facial expressions in real time, and matches them with the animation generated by lip sync, making every expression of the virtual character in the interaction more realistic and credible. In the final output video, the digital human's lip movements are smooth and the facial expressions are delicate, which greatly enhances the user's visual experience and interactive immersion, achieving a realistic and natural interaction effect.

[0094] Third, customized background

[0095] The digital human real-time interactive system 100 has the ability to support multiple virtual backgrounds, meeting the needs of different application scenarios and greatly enhancing the immersiveness of user interaction. Through the deep integration of image synthesis technology and real-time rendering engine, the system can seamlessly integrate model videos with static images or dynamic video backgrounds to create an interactive experience with rich visual impact.

[0096] The image rendering module not only supports traditional static backgrounds, but can also load and render dynamic backgrounds in real time, which enables digital humans to adapt to diverse visual needs in multiple fields such as social, education, entertainment, and marketing. In social scenes, you can choose virtual environments of different styles, such as modern cities, natural scenery, or fantasy scenes, to enhance the fun and personalization of interaction. In the field of education, the system can provide corresponding professional backgrounds such as laboratories, classrooms, or conference rooms based on the course theme, thereby enhancing the intuitiveness and professionalism of learning.

[0097] In order to achieve the flexibility of multiple backgrounds, the system uses background separation and dynamic synthesis technology, which can quickly switch the background during user interaction without affecting the performance of the digital human. This technology accurately identifies background and foreground elements through a deep learning model, allowing the virtual character to maintain smooth animation effects in different backgrounds, while ensuring that the lighting and color of the background are consistent with the performance of the character, enhancing the overall sense of reality.

[0098] The system supports background adaptive algorithms, which can automatically adjust background effects according to ambient lighting, user actions and voice commands to enhance the user's immersive experience. The application of this technology enables users to feel more natural and realistic environmental changes when interacting with digital humans, thereby increasing their sense of participation and enthusiasm for interaction.

[0099] In summary, by supporting multiple virtual backgrounds, this system not only enhances the immersiveness of interaction, but also greatly enriches the diversity of application scenarios and promotes the application and development of virtual human technology in various fields.

[0100] In addition, system 100 also has strong scalability and can support the integration of third-party services through a plug-in mechanism, such as online education, virtual customer service, game interaction and other application fields to meet the needs of various scenarios.

[0101] Figure 4 A flow chart of a digital human real-time interaction method 400 according to some embodiments of the present application is shown. According to the implementation of the present application, the method 400 is suitable for execution in the cloud server 120. When the user inputs a voice through the terminal device 110, the terminal device 110 uploads the user's input voice to the cloud server 120, which executes the method 400 to generate a response video. Afterwards, the cloud server 120 returns the response video to the terminal device 110, and the terminal device 110 displays the response video to the user.

[0102] refer to Figure 4 The method 400 starts at step S410. In step S410, the speech input by the user is converted into text.

[0103] Then in S420, the text is semantically analyzed to determine the user's intention, and based on the user's intention, a corresponding reply text is generated.

[0104] According to the implementation of the present application, a pre-trained language model is used to build a model for multi-round dialogues to process the user's text and generate corresponding reply text. In some embodiments, a Transformer-based language model, such as GPT, is used to dynamically adjust the reply text based on the user's historical dialogue records and context information to improve the processing ability of complex queries.

[0105] In addition, when generating the reply text, information can be obtained from pre-generated knowledge bases and external knowledge bases.

[0106] Then in S430, the generated reply text is converted into reply voice.

[0107] Then in S440, based on the audio-visual synchronization model, the reply voice and the pre-generated continuous image frames are processed to generate audio-driven image frames as the first video data, wherein the lip shape of the first video data is synchronized with the reply voice, and the continuous image frames contain a digital human image.

[0108] The audio-visual synchronization model is based on Wav2Lip, a deep learning model based on convolutional neural network (CNN) that is specifically designed for real-time matching of audio and lip movements.

[0109] In order to ensure real-time performance, this embodiment further optimizes the reasoning process of Wav2Lip. The optimization is mainly to capture the facial lip shape, accurately lock the lip shape area of ​​the object (i.e., digital human), and record and analyze the characteristics of the lip shape to achieve rapid simulation of human speech. At the same time, the generation efficiency is improved through parallel computing technology, ensuring that the lip shape and voice of the digital human are synchronized and natural when playing audio, showing high-quality interactive effects.

[0110] According to this embodiment, method 400 also includes the step of pre-generating continuous image frames. Specifically, the original video data of the object indicating the digital human image is collected, wherein the original video data includes the body movements and facial expressions of the object; the original video data is processed to obtain continuous image frames containing only the object. In some embodiments, the processing of the original video data includes: using matting processing to obtain transparent channel video data containing only the object; after matting, the edge jagged effect is eliminated by the edge sharpening algorithm (Edge Sharpening) to ensure the clarity and natural transition effect of the object; and the timing consistency correction is performed to ensure the coherence and stability of the body movements and facial expressions of the object in the generated continuous image frames.

[0111] Then, in S450, the first video data is merged with the virtual background, and the merged video frames are processed in real time to generate a response video to the user's voice.

[0112] According to the method 400 of the present application, a reply voice is automatically generated for the voice input by the user in combination with natural speech recognition, natural language processing, etc. In addition, through a deep learning model based on Wav2Lip, real-time lip shape changes of the digital human are dynamically generated, and combined with the generated reply voice, efficient voice and lip shape synchronization interaction is achieved.

[0113] It should be noted that the description of method 400 and the aforementioned system 100 complement each other, and the related parts will not be repeated here.

[0114] The system 100 of the present application can be implemented by one or more computing devices. The computing device can be any device with storage and computing capabilities, such as a server, a workstation, etc., or a personal computer such as a desktop computer or a notebook computer, or a terminal device such as a mobile phone, a tablet computer, a smart wearable device, an Internet of Things device, etc., but is not limited thereto.

[0115] Figure 5 FIG. 5 is a schematic diagram of a computing device 500 according to some embodiments of the present application. It should be noted that: Figure 5 The computing device 500 shown is only an example. In practice, the computing device used to implement the method of the present application can be any type of device, and its hardware configuration can be different from the above. Figure 5 The computing device 500 shown is the same as Figure 5 The computing device 500 shown is different. In practice, the computing device used to implement the embodiments of the present application can be Figure 5 The hardware components of the computing device 500 shown are increased or deleted, and the present application does not limit the specific hardware configuration of the computing device.

[0116] like Figure 5 In a basic configuration, the computing device 500 includes at least one processing unit 502 and a system memory 504. According to one aspect, depending on the configuration and type of the computing device, the processing unit 502 can be implemented as a processor. The system memory 504 includes, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, the system memory 504 includes an operating system 505 and program modules 506.

[0117] According to one aspect, operating system 505 is suitable, for example, for controlling the operation of computing device 500. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other application programs, and are not limited to any particular application or system. Figure 5 This basic configuration is illustrated in FIG. 5 by those components within dashed line 508. According to one aspect, computing device 500 has additional features or functionality. For example, according to one aspect, computing device 500 includes additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 5 , which is illustrated by removable storage 509 and non-removable storage 510.

[0118] As stated above, according to one aspect, program modules are stored in the system memory 504. According to one aspect, the program modules may include one or more application programs, and the present application does not limit the types of application programs, for example, the application programs may include: email and contact applications, word processing applications, spreadsheet applications, database applications, slide show applications, painting or computer-aided applications, web browser applications, etc.

[0119] According to one aspect, the examples may be practiced on a circuit comprising discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or a single chip containing electronic components or a microprocessor. Figure 5 Each or many components shown in can be integrated in a system on chip (SOC) on a single integrated circuit to practice examples. According to one aspect, such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") into a chip substrate as a single integrated circuit. When operated via SOC, the functions described in the present application can be operated via a dedicated logic integrated with other components of the computing device 500 on a single integrated circuit (chip). Other technologies that can perform logical operations (such as AND, OR, and NOT) can also be used to practice embodiments of the present application, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present application can be practiced in a general-purpose computer or in any other circuit or system.

[0120] According to one aspect, the computing device 500 may also have one or more input devices 512, such as a keyboard, a mouse, a pen, a voice input device, a touch input device, a VR motion capture input device, etc. Output devices 514 may also be included, such as a display, a speaker, a printer, etc. The aforementioned devices are examples and other devices may also be used. The computing device 500 may include one or more communication connections 516 that allow communication with other computing devices 518. Examples of suitable communication connections 516 include, but are not limited to: RF transmitters, receivers and / or transceiver circuits; Universal Serial Bus (USB), parallel and / or serial ports.

[0121] The term computer-readable media as used in this application includes computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer-readable instructions, data structures, or program modules). System memory 504, removable storage 509, and non-removable storage 510 are all examples of computer storage media (i.e., memory storage). Computer storage media may include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other products that can be used to store information and can be accessed by computing device 500. According to one aspect, any such computer storage media may be part of computing device 500. Computer storage media do not include carrier waves or other propagated data signals.

[0122] According to one aspect, communication media is implemented by computer readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier wave or other transport mechanism), and includes any information delivery media. According to one aspect, the term "modulated data signal" describes a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0123] In an embodiment according to the present application, the computing device 500 is configured to execute the digital human real-time interaction method 400 according to the present application. The program module 506 includes a plurality of program instructions for executing the method 400, and these program instructions can instruct the processing unit 502 to execute the method 400 of the present application.

[0124] Optionally, a real-time interaction method for digital humans is suitable for execution in a cloud server, the method comprising: converting voice input by a user into text; performing semantic analysis on the text to determine user intent, and generating corresponding reply text based on the user intent; converting the generated reply text into reply voice; based on an audio-visual synchronization model, processing the reply voice and pre-generated continuous image frames to generate audio-driven image frames as first video data, wherein the lip shape of the first video data is synchronized with the reply voice, and the continuous image frames contain a digital human image; fusing the first video data with a virtual background, and processing the fused video frames in real time to generate a response video to the user voice.

[0125] Optionally, the method further comprises the step of pre-generating continuous image frames: collecting original video data of an object indicating a digital human image, wherein the original video data comprises body movements and facial expressions of the object; and processing the original video data to obtain continuous image frames containing only the object.

[0126] Optionally, a computing device comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method described above.

[0127] Optionally, a computer-readable storage medium stores one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device performs the method as described above.

[0128] Optionally, a computer program product comprises a computer program / instructions, wherein the method is implemented when the computer program / instructions are executed by a processor.

[0129] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present application, or certain aspects or portions of the method and apparatus of the present application may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present application.

[0130] In the case where the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the method of the present application according to the instructions in the program code stored in the memory.

[0131] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium. Any combination of the above is also included within the scope of readable media.

[0132] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present application. According to the above description, it is obvious to construct the structure required for such systems. In addition, the present application is not directed to any specific programming language either. It should be understood that the content of the present application described herein can be realized by various programming languages, and the description of the specific language above is to disclose the preferred implementation of the present application.

[0133] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0134] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0135] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition they can be divided into multiple submodules or subunits or subassemblies. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise clearly stated, each feature disclosed in this specification can be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0136] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present application and to form different embodiments.

[0137] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices that are used to implement the functions performed by the elements for the purpose of implementing the disclosure.

[0138] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely refers to different instances of similar objects and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner. In addition, the quantifier "plurality" means "two" and / or "more than two."

[0139] Although the present application is described according to a limited number of embodiments, it will be appreciated by those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the present application thus described. In addition, it should be noted that the language used in this specification is selected primarily for the purpose of readability and teaching, rather than for the purpose of explaining or limiting the subject matter of the present application. Therefore, many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. For the scope of the present application, the disclosure made to the present application is illustrative, not restrictive, and the scope of the present application is defined by the appended claims.

Claims

1. A digital human real-time interaction system, comprising: A terminal device, comprising an audio input module and a video output module, wherein the terminal device is adapted to receive a voice input by a user via the audio input module and transmit the voice to a cloud server, and is also adapted to output a response video to the user's voice via the video output module; A cloud server is adapted to process the voice input by the user in real time to generate the response video and transmit the response video to the terminal device, wherein the cloud server comprises: a video acquisition and processing module adapted to acquire raw video data of an object indicating a digital human image, wherein the raw video data includes body movements and facial expressions of the object, and to generate continuous image frames containing only the object; An intelligent question-answering and speech generation module, adapted to convert the speech input by the user into text; Performing semantic analysis on the text, understanding the user's intention, and generating corresponding reply text using a language model; and converting the generated reply text into reply speech; an audio processing module, adapted to process the reply speech and the continuous image frames based on an audio-visual synchronization model to generate audio-driven image frames as first video data, wherein the lip shape of the first video data is synchronized with the reply speech; The image rendering module is adapted to fuse the first video data with a virtual background and perform real-time processing on the fused video frames to generate the response video, wherein the virtual background includes a static image, a dynamic video, and a three-dimensional scene.

2. The system of claim 1, wherein: In the cloud server, the video acquisition and processing module includes: A video acquisition unit, adapted to acquire original video data using a green screen background, and capture the body movements and facial expressions of the subject to obtain original video data; A video processing unit, adapted to process the original video data to obtain the continuous image frames, comprising: Using an algorithm based on chromaticity difference, the green screen background in the original video data is removed to obtain transparent channel video data containing the object. sharpening the edges in the transparent channel video data to obtain processed transparent channel video data, The processed transparent channel video data is corrected to generate continuous image frames with consistent time sequence.

3. The system of claim 1 or 2, wherein: In the cloud server, The audio processing module is also suitable for training and generating an audio-visual synchronization model based on a neural network. The audio-visual synchronization model is suitable for extracting features of the input reply speech, generating spectral features corresponding to the reply speech, analyzing the input continuous image frames, extracting visual features of the object under different facial expressions and lip states, and matching the spectral features and the visual features to generate a video with synchronized speech and lip shape.

4. The system of claim 3, wherein: In the cloud server, the audio processing module includes: a text conversion unit adapted to convert natural language text into audio data; The training unit is suitable for generating training samples using the audio data and its corresponding real video data, and using a pre-trained model to train and generate an audio-visual synchronization model through the training samples, wherein the pre-trained model is a deep learning model based on Wav2Lip.

5. The system according to any one of claims 1 to 4, wherein: In the cloud server, the intelligent question-answering and speech generation module includes: An automatic speech recognition unit, adapted to recognize the speech input by the user using an acoustic model based on a deep neural network, and obtain a text corresponding to the speech; A natural language processing unit, adapted to use a language model to deeply understand the text, determine the user's intention, and generate a reply text for the user's input voice according to the user's intention; The text-to-speech unit is adapted to convert the reply text into reply speech through a neural network speech synthesis method.

6. The system of claim 5, wherein: In the cloud server, the natural language processing unit includes: A text preprocessing subunit, adapted to process the text using a word segmentation algorithm to obtain a plurality of word segments; The intention recognition subunit is adapted to perform semantic analysis and context understanding on multiple word segments of the current text and the conversation history through a conversation state tracking algorithm to determine the current conversation state, wherein the conversation state includes the user intention; The dialogue management subunit is adapted to obtain required information from a relevant knowledge base, generate an initial reply text, and perform semantic and grammatical checks on the initial reply text to obtain the reply text.

7. The system of claim 6, wherein: In the cloud server, The intelligent question-answering and speech generation module is also suitable for accessing an external knowledge base through an interface.

8. The system of any one of claims 1 to 7, wherein: In the cloud server, the image rendering module is further adapted to: Preprocessing the virtual background according to the actual application scenario. If the virtual background is a static background, the preprocessing includes adjusting the resolution, color space and frame rate. If the virtual background is a dynamic background, the preprocessing includes using a time series-based synchronous rendering technology to ensure the consistency of the virtual background and the object's movements. The preprocessed virtual background is integrated with the first video data, including: dynamically adjusting the position and intensity of the light source in the preprocessed virtual background by physically based rendering technology, and when the preprocessed virtual background is a three-dimensional scene, using a depth map-based rendering technology to add a sense of hierarchy to the object and the virtual background to achieve a realistic representation of spatial depth; Use a real-time rendering engine to process the fused video images in real time and output high-resolution responsive videos.

9. The system of claim 8, wherein: The image rendering module is also suitable for automatically adjusting the virtual background according to ambient lighting, user actions and voice commands.

10. The system of any one of claims 1 to 9, wherein: The terminal device further includes: The audio signal processing module is adapted to perform preliminary processing on the voice input by the user.

Images