Digital human real-time dialogue method and system based on image cloning, terminal and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a digital human real-time dialogue method and system based on image cloning, a terminal and a medium, and the method comprises the steps: segmenting an original user video into a mouth closing silent video and a non-silent video; intercepting images of a face area in the non-silent video frame by frame and synthesizing the images into a training video; training the image cloning model based on the training video; the answer text generated by the large language model and the voice synthesis model is converted into audio data, a face image sequence matched with the audio data is generated in combination with the trained image clone model, the face image sequence is synthesized into a non-silent video and then pushed out, and if the duration of the non-silent video is not enough, the non-silent video is pushed out. And if not, circularly pushing the mouth closing silence video to the front end in a non-conversation state by adopting a positive-sequence and reverse-sequence loop playing strategy. Through a specific video preprocessing process and a playing mode, the continuity problem of a long dialogue scene is improved, and the naturalness of a digital human in a non-dialogue state is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system, terminal and medium for real-time dialogue between digital humans based on image cloning. Background Art

[0002] With the rapid development of artificial intelligence technology, the application boundaries of digital human technology in many fields are being continuously broadened, and it has shown more and more extensive application potential in virtual social networking, intelligent customer service, education and training, and film and television production. Digital human technology has successfully achieved a seamless and natural interactive experience with users by highly simulating the appearance characteristics, delicate expression changes, and natural and smooth voice of real humans. This breakthrough has greatly improved the overall sense of participation and satisfaction of users.

[0003] Under the current technical framework, the generation of digital humans mainly relies on a series of advanced and complex technical means. Among them, face detection and reconstruction technology based on computer vision can accurately capture and analyze the subtle features of the face, thereby constructing a highly realistic digital face model; the use of natural language processing technology enables digital humans to understand and generate text content that conforms to the context, promoting more fluent and intelligent dialogue and communication; and audio generation technology based on speech synthesis allows digital humans to have voice expressions close to real people, further enhancing the realism of their interactions.

[0004] However, despite the significant progress made in digital human technology, it still faces many challenges and problems in its actual application, such as: 1. In long dialogue scenarios, the movements of digital human videos are not coherent and natural; 2. The naturalness and authenticity of digital humans in non-dialogue states are quite different from those of real people, which affects the user's immersive experience and limits the in-depth application of digital human technology in more scenarios. Summary of the invention

[0005] In order to solve the technical problem that the movements of digital human videos in long dialogue scenarios are not coherent and natural, and the naturalness and authenticity of digital humans in non-dialogue states are quite different from those of real people, which affects the user's immersive experience, the present invention provides, on the one hand, a digital human real-time dialogue method based on image cloning, and on the other hand, a digital human real-time dialogue system based on image cloning, a terminal and a medium.

[0006] To achieve the above-mentioned purpose, the technical solution adopted by the digital human real-time dialogue method based on image cloning in the present invention is: Obtaining an original user video; dividing and arranging the original user video into a closed-mouth silent video and a non-silent video based on the user's mouth state in the original user video; Calculate the bounding box of the face area in the non-silent video based on the face detection algorithm and record the coordinate information of the bounding box; capture the image of the face area in the non-silent video frame by frame based on the bounding box; synthesize the training video using the image of the face area; train the image cloning model based on the training video to obtain the trained image cloning model; Acquire the conversation text input by the user in real time, input the conversation text into the large language model to generate the answer text, and use the speech synthesis model to convert the answer text into audio data; The audio data is imported into the trained image cloning model to generate a facial image sequence that matches the audio data; the facial image sequence is synthesized into the non-silent video according to the coordinate information of the bounding box to obtain a conversation video; when the duration of playing the facial image sequence exceeds the duration of the non-silent video, the duration of the non-silent video is extended by using a forward and reverse loop playback method; The audio data and the dialogue video are pushed to the front end synchronously; when there is no dialogue, the silent video of closing the mouth is pushed to the front end in a loop.

[0007] By adopting the above technical solution, this application improves the coherence and naturalness of digital human videos in long dialogue scenarios through a specific video preprocessing process and a specific playback method, improves the naturalness of digital humans in non-dialogue states, reduces the difference between digital humans and real people, realizes natural dialogue interaction between users and digital humans, and improves user experience. The specific video preprocessing process includes: dividing and organizing the original user video into closed-mouth silent videos and non-silent videos, training the non-silent videos, and the closed-mouth silent videos do not participate in the training. In the absence of dialogue, the closed-mouth silent videos are pushed to the front end in a loop. The specific playback method includes: when the duration of playing the facial image sequence exceeds the duration of the non-silent video, the duration of the non-silent video is extended by using a forward and reverse loop playback method.

[0008] As a preferred implementation of the digital human real-time dialogue method based on image cloning, Based on the face detection algorithm, the bounding box of the face area in the non-silent video is calculated, and the coordinate information of the bounding box is recorded, including: The position of the face area in each frame of the non-silent video is calculated based on the face detection algorithm, the position of the bounding box that can contain the face area in each frame is calculated based on the position of the face area in each frame, and the coordinate information of the bounding box is recorded based on the position of the bounding box.

[0009] By adopting the above technical solution, the bounding box can frame the face area in each frame of the non-silent video. The same bounding box is used to capture the image of the face area in each frame of the non-silent video, which can ensure that the captured image of the face area contains the complete face.

[0010] As a preferred implementation of the digital human real-time dialogue method based on image cloning, the image cloning model is an ER-NeRF model or a SyncTalk model.

[0011] As a preferred implementation of the method for real-time conversation of digital human based on image cloning, after the audio data is imported into the trained image cloning model and a facial image sequence matching the audio data is generated, the following steps are also included: Smooth the face image sequence to eliminate inter-frame jitter.

[0012] The above technical solution makes the face image sequence more natural and coherent when played frame by frame, improving the user's immersive experience. As a preferred implementation of the method for real-time conversation of digital human based on image cloning, the forward and reverse order loop playback method includes the following steps: When the non-silent video is played to the end, the non-silent video is played in reverse. When the non-silent video is played in reverse to the starting position of the non-silent video, the non-silent video is played forward, and the cycle is repeated until the duration of the face image sequence is reached.

[0013] By adopting the above technical solution, the duration of the non-silent video is extended, with the aim of obtaining a video background with a coherent action state, thereby improving the coherence of the user's action state in the conversation video.

[0014] As a preferred implementation of the real-time conversation method of digital human based on image cloning, the audio data and conversation video are pushed to the front end synchronously; in the absence of conversation, the closed-mouth silent video is pushed to the front end in a loop, including: The audio data and conversation video are pushed to the front end synchronously through WebRTC technology; when there is no conversation, the silent video is pushed to the front end in a loop through WebRTC technology.

[0015] As a preferred implementation of the digital human real-time dialogue method based on image cloning, the original user video contains a full-body or half-body picture, the user's movement range does not exceed the shoulder range, the original user video is not less than 5 minutes, and the duration of the closed-mouth silent video is not less than 10 seconds.

[0016] The adoption of the above technical solution can improve the authenticity and accuracy of the conversation video output by the trained image cloning model.

[0017] On the other hand, the technical solution adopted by the digital human real-time dialogue system based on image cloning in the present invention is: The digital human real-time dialogue system based on image cloning includes: Video acquisition module, the video acquisition module is configured as Based on the mouth state of the user in the original user video, the original user video is segmented and sorted into closed-mouth silent videos and non-silent videos; A face processing module, the face processing module is configured to calculate a bounding box of a face area in a non-silent video based on a face detection algorithm, and record coordinate information of the bounding box; capture an image of the face area in the non-silent video frame by frame based on the bounding box; and synthesize a training video using the image of the face area; A model training module, the model training module is configured to train the image cloning model based on the training video to obtain a trained image cloning model; A dialogue processing module, the dialogue processing module is configured to obtain dialogue text input by the user in real time, input the dialogue text into the large language model to generate answer text, and use the speech synthesis model to convert the answer text into audio data; The picture generation module is configured to import the audio data into the trained image cloning model, generate a facial image sequence matching the audio data, synthesize the facial image sequence into the original user video based on the coordinate information of the facial region in each frame of the original user video, and when the duration of playing the facial image sequence exceeds the duration of the original user video, extend the duration of the original user video by playing it in a forward and reverse order loop to obtain a conversation video; The real-time push module is configured to push the audio data and the conversation video to the front end synchronously; in the absence of conversation, the silent video of closing the mouth is pushed to the front end in a loop.

[0018] On the other hand, a technical solution adopted by a terminal in the present invention is: A terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for real-time conversation of digital human based on image cloning are implemented.

[0019] On the other hand, a technical solution adopted by a medium in the present invention is: A medium stores a computer program, which, when executed by a processor, implements the steps of the method for real-time conversation of digital human based on image cloning as described above.

[0020] The beneficial effects of the present invention include: This application improves the coherence and naturalness of digital human videos in long dialogue scenarios through specific video preprocessing processes and specific playback methods, improves the naturalness of digital humans in non-dialogue states, reduces the differences between digital humans and real people, enables natural dialogue interaction between users and digital humans, and improves user experience. The specific video preprocessing process includes: dividing and organizing the original user video into closed-mouth silent videos and non-silent videos, training the non-silent videos, and not participating in the training. In the absence of dialogue, the closed-mouth silent videos are pushed to the front end in a loop. The specific playback method includes: when the duration of playing the facial image sequence exceeds the duration of the non-silent video, the duration of the non-silent video is extended by using a forward and reverse loop playback method. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0022] Figure 1 It is a schematic flow chart of a digital human real-time dialogue method based on image cloning in a specific embodiment of the present invention; Figure 2 It is a schematic block diagram of a digital human real-time dialogue system based on image cloning in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0023] Under the current technical framework, the generation of digital humans mainly relies on a series of advanced and complex technical means. Among them, face detection and reconstruction technology based on computer vision can accurately capture and analyze the subtle features of the face, thereby constructing a highly realistic digital face model; the use of natural language processing technology enables digital humans to understand and generate text content that conforms to the context, promoting more fluent and intelligent dialogue and communication; and audio generation technology based on speech synthesis allows digital humans to have voice expressions close to real people, further enhancing the realism of their interactions.

[0024] However, despite the significant progress made in digital human technology, it still faces many challenges and problems in its actual application, such as: 1. In long dialogue scenarios, the movements of digital human videos are not coherent and natural; 2. The naturalness and authenticity of digital humans in non-dialogue states are quite different from those of real people, which affects the user's immersive experience and limits the in-depth application of digital human technology in more scenarios.

[0025] How to solve the problem of incoherent and unnatural movements in digital human videos in long dialogue scenarios? The naturalness and authenticity of digital humans in non-dialogue states are quite different from those of real people, which affects the user's immersive experience. This has become a technical problem that needs to be solved urgently.

[0026] Based on this, this embodiment proposes a real-time conversation method for digital humans based on image cloning, which can improve the continuity and naturalness of digital human videos in long conversation scenarios through specific video preprocessing processes and specific playback methods, enhance the naturalness of digital humans in non-conversation states, and reduce the differences between digital humans and real people.

[0027] The following is a detailed description of the digital human real-time dialogue method based on image cloning involved in the present application. For the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0028] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" etc. that appear in different places in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Reference Figure 1 This application proposes a digital human real-time dialogue method based on image cloning, comprising the following steps: S1. Obtain original user video; S2, segmenting the original user video into closed-mouth silent videos and non-silent videos based on the mouth state of the user in the original user video; S3, calculating the bounding box of the face area in the non-silent video based on the face detection algorithm, and recording the coordinate information of the bounding box; and capturing the image of the face area in the non-silent video frame by frame based on the bounding box; S4, using the image of the face area to synthesize the training video; S5, training the image cloning model based on the training video to obtain a trained image cloning model; S6. Acquire the dialogue text input by the user in real time, input the dialogue text into the large language model to generate a response text, and use the speech synthesis model to convert the response text into audio data; S7, importing the audio data into the trained image cloning model to generate a facial image sequence matching the audio data; synthesizing the facial image sequence into the non-silent video according to the coordinate information of the bounding box to obtain a conversation video, and when the duration of playing the facial image sequence exceeds the duration of the non-silent video, a forward and reverse loop playback method is used to extend the duration of the non-silent video; S8. Push the audio data and conversation video to the front end synchronously; when there is no conversation, push the silent video to the front end in a loop.

[0031] This embodiment significantly improves the coherence and natural fluency of the digital human video in long dialogue scenarios through a carefully designed specific video preprocessing process and an innovative specific playback method, and enhances the naturalness of the digital human in a non-dialogue state, so that the digital human can also show a natural state that is extremely similar to a real person in non-direct dialogue moments such as silence, thinking, or waiting for instructions, greatly narrowing the difference in behavioral performance with real humans, so that the digital human is close to the authenticity of the original user video whether in dialogue or silence.

[0032] This embodiment not only realizes a more natural and smooth dialogue interaction between the user and the digital human, allowing the user to feel as if he or she is communicating with a real person, but also fundamentally improves the user experience and enhances the user's acceptance and satisfaction with the digital human technology.

[0033] The specific video preprocessing process includes: dividing the original user video into closed-mouth silent videos and non-silent videos, training the non-silent videos, and not participating in the training of closed-mouth silent videos. In the absence of dialogue, the closed-mouth silent videos are pushed to the front end in a loop.

[0034] The specific playback method includes: when the duration of playing the facial image sequence exceeds the duration of the non-silent video, a forward and reverse loop playback method is used to extend the duration of the non-silent video.

[0035] This embodiment only trains training videos synthesized from images of face areas captured from non-silent videos. It does not train all original user videos, nor does it train character images containing face and body parts in the videos. This can reduce the training time of the image cloning model, improve the output efficiency of the image cloning model, and further improve the efficiency of real-time conversations with digital humans.

[0036] In a specific embodiment, the step S2 of calculating the bounding box of the face area in the non-silent video based on the face detection algorithm and recording the coordinate information of the bounding box includes: calculating the position of the face area in each frame of the non-silent video based on the face detection algorithm, calculating the position of the bounding box that can contain the face area in each frame based on the position of the face area in each frame, and recording the coordinate information of the bounding box based on the position of the bounding box. Thus, the bounding box can frame the face area in each frame of the non-silent video, and the same bounding box is used to capture the image of the face area on each frame of the non-silent video, which can ensure that the captured image of the face area contains a complete face.

[0037] In some specific embodiments, the image cloning model is an ER-NeRF model or a SyncTalk model or other image cloning models based on neural radiation fields.

[0038] Among them, NeRF (Neural Radiance Fields) is a three-dimensional scene reconstruction method based on neural networks, which can restore realistic three-dimensional scenes from two-dimensional image data.

[0039] The ER-NeRF model is an optimization and improvement on NeRF, making it more suitable for the generation of digital humans in real-time conversations. By introducing a compact and expressive three-plane hash representation and a regional attention module, the ER-NeRF model can generate digital humans with higher fidelity and audio lip synchronization. Compared with existing methods, the digital humans generated by the ER-NeRF model are more realistic in details.

[0040] The SyncTalk model effectively maintains the subject identity based on NeRF's approach and enhances the synchronization and realism of talking head synthesis. The SyncTalk model uses a facial synchronization controller to align lip movements with speech and innovatively uses a 3D facial blend shape model to capture accurate facial expressions.

[0041] In some specific embodiments, the above-mentioned speech synthesis model is a ChatTTS model or a CosyVoice model or other speech synthesis models that support voice cloning.

[0042] The ChatTTS model is a text-to-speech (TTS) model designed for conversational environments, supporting Chinese and English speech synthesis. It is trained on large-scale Chinese and English corpora to generate natural and fluent speech output. The model optimizes conversational tasks, ensures the naturalness and fluency of speech synthesis, supports multiple speakers, has the ability to accurately control prosodic features, and can achieve more realistic speech effects. The main functions are: 1. Text to speech: ChatTTS can convert input text into natural and fluent speech in real time, suitable for multiple language environments.

[0043] 2. Multi-language support: In addition to Chinese, ChatTTS can also process English text to serve a wider user group.

[0044] 3. Emotion and rhythm adjustment: In addition to the basic text-to-speech function, ChatTTS can also adjust the emotional color and rhythmic characteristics of the voice according to the text content, such as speaking speed and pauses, making the voice closer to natural human communication.

[0045] 4. Voice role selection: Users can choose the most suitable voice from multiple preset voice roles according to scene requirements to enhance the personalization and expressiveness of voice.

[0046] 5. Interactive Web Interface: Users can easily input text and get voice output through an intuitive web interface without any programming experience.

[0047] 6. Real-time voice interaction: ChatTTS supports instant speech synthesis, which is very suitable for dialogue systems and interactive applications that require fast feedback.

[0048] 7. Voice file export: Users can export synthesized voice into common audio formats for subsequent editing and sharing.

[0049] 8. Integration and compatibility: ChatTTS can be seamlessly integrated into various platforms and applications, including web applications, mobile applications, and desktop software.

[0050] 9. Emotional tagging system: supports inserting emotional tags into text. Users can finely control the emotional expression of speech, such as simulating laughter by inserting [laugh].

[0051] The CosyVoice model is a multilingual speech generation model that provides full-stack capabilities for reasoning, training, and deployment. The CosyVoice model mainly includes the following features: 1. Multi-language support: Able to generate speech in multiple languages, including Chinese, English, Japanese, Cantonese and Korean.

[0052] 2. Multiple reasoning modes: It supports multiple modes such as zero-sample reasoning, cross-language reasoning, and command reasoning. It only needs a few seconds of voice samples to imitate the voice of the target speaker. It can use voice samples of one language to synthesize speech in another language, and supports controlling the emotion, tone and other details of the voice through commands.

[0053] 3. Complete training and deployment support: Provides full-process support from data preparation, model training to model deployment.

[0054] In some embodiments, in step S7, when the audio data is imported into the trained image cloning model to generate a facial image sequence matching the audio data, the following steps are also included: The facial image sequence is smoothed to eliminate jitter between frames, making the facial image sequence more natural and coherent when played frame by frame, thus improving the user's immersive experience.

[0055] In this embodiment, when the facial image sequence is synthesized into the non-silent video according to the coordinate information of the bounding box, the facial image sequence is naturally integrated with the background other than the face in the non-silent video.

[0056] In some embodiments, the forward and reverse order loop playback method described in step S7 includes the following steps: When the non-silent video is played to the end, the non-silent video is played in reverse. When the non-silent video is played in reverse to the starting position of the non-silent video, the non-silent video is played forward, and the cycle repeats until the duration of the face image sequence is reached. In this way, the duration of the non-silent video is extended, and the purpose is to obtain a video background with a coherent action state, thereby improving the coherence of the user's action state in the conversation video.

[0057] Furthermore, when the conversation video is generated in step S7, it is preferably necessary to perform synchronization calibration with the audio data to ensure synchronization of the audio and the video.

[0058] In some embodiments, in step S8, the audio data and the conversation video are synchronously pushed to the front end; in the absence of a conversation, the closed-mouth silent video is cyclically pushed to the front end, including: The audio data and conversation video are pushed to the front end synchronously through WebRTC technology; when there is no conversation, the silent video is pushed to the front end in a loop through WebRTC technology.

[0059] Among them, RTC (Real-time Communications) is an abbreviation for real-time audio and video. WebRTC (Web Real-Time Communications) is a part of RTC, which is a communication technology that allows users to achieve real-time audio and video communication using their own traffic.

[0060] Furthermore, the process of pushing through WebRTC technology also includes: Dynamically adjust the quality of pushed videos according to network conditions; Real-time monitoring of audio and video synchronization status; Automatically handle network latency and packet loss.

[0061] Thereby, the quality of pushed videos is further improved, the synchronization of audio and video is further guaranteed, and the pushed videos are further made complete and without lag.

[0062] In this embodiment, in order to improve the authenticity and accuracy of the conversation video, the original user video preferably needs to include a full-body or half-body picture, the user's movement range does not exceed the shoulder range, the user's movement does not include directional movements (movements with obvious meanings, such as fingers making specific shapes), the original user video is not less than 5 minutes, and the length of the closed-mouth silent video is not less than 10 seconds.

[0063] Furthermore, the shooting environment of the user's original video is preferably an environment with sufficient light and a single background; the shooting angle is preferably mainly frontal, and the left and right deflection angles do not exceed 15 degrees; the shooting distance requirement is preferably that the proportion of the characters in the picture is appropriate, and the area above the shoulders is kept clearly visible when shooting half-body; expressions include daily expressions such as natural speaking and smiling.

[0064] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0065] The digital human real-time dialogue method based on image cloning disclosed in this embodiment can be applied to a variety of scenarios, such as: 1. Virtual social networking: Users can create their own virtual images and interact with other users’ digital people on social media platforms, breaking geographical restrictions and providing more personalized expression space.

[0066] 2. Intelligent customer service: Enterprises can customize their own digital customer service to provide users with 24-hour uninterrupted online services and improve customer satisfaction and loyalty.

[0067] 3. Education and training: Educational institutions can use digital human technology to conduct remote teaching, personalized tutoring, etc., to provide students with a more vivid and intuitive learning experience.

[0068] 4. Film and television production: Digital human technology can be applied to the production of movies, animations and other film and television works, bringing more realistic and shocking visual enjoyment to the audience.

[0069] The application scenarios of this embodiment include but are not limited to the above examples. This embodiment can break through the constraints of the physical world, realize communication and interaction across time and space, provide highly personalized services and experiences, meet the diverse needs of users, reduce dependence on human resources, and improve service quality and efficiency.

[0070] like Figure 2As shown, the following is an embodiment of a digital human real-time dialogue system based on image cloning provided by an embodiment of the present disclosure. The digital human real-time dialogue system based on image cloning and the digital human real-time dialogue method based on image cloning in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the digital human real-time dialogue system based on image cloning, reference can be made to the above-mentioned embodiment of the digital human real-time dialogue method based on image cloning.

[0071] A digital human real-time dialogue system based on image cloning includes a video acquisition module, a video processing module, a face processing module, a model training module, a dialogue processing module, a picture generation module and a real-time push module, wherein: Video acquisition module, the video acquisition module is configured as Based on the mouth state of the user in the original user video, the original user video is segmented and sorted into closed-mouth silent videos and non-silent videos; A face processing module, the face processing module is configured to calculate a bounding box of a face area in a non-silent video based on a face detection algorithm, and record coordinate information of the bounding box; capture an image of the face area in the non-silent video frame by frame based on the bounding box; and synthesize a training video using the image of the face area; A model training module, the model training module is configured to train the image cloning model based on the training video to obtain a trained image cloning model; A dialogue processing module, the dialogue processing module is configured to obtain dialogue text input by the user in real time, input the dialogue text into the large language model to generate answer text, and use the speech synthesis model to convert the answer text into audio data; The picture generation module is configured to import the audio data into the trained image cloning model, generate a facial image sequence matching the audio data, synthesize the facial image sequence into the original user video based on the coordinate information of the facial region in each frame of the original user video, and when the duration of playing the facial image sequence exceeds the duration of the original user video, extend the duration of the original user video by playing it in a forward and reverse order loop to obtain a conversation video; The real-time push module is configured to push the audio data and the conversation video to the front end synchronously; in the absence of conversation, the silent video of closing the mouth is pushed to the front end in a loop.

[0072] An embodiment of the present application also proposes a terminal, including a memory, a processor, a communication unit, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, steps of a method for real-time conversation of a digital human based on image cloning are implemented; the memory, the processor, and the communication unit communicate via one or more buses.

[0073] The processor may include one or more processing units, for example, the processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0074] The processor can be the nerve center and command center of the terminal. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0075] The memory is used to store the execution instructions of the processor, and the memory can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory are executed by the processor, the terminal can execute some or all of the steps in the above-mentioned embodiment of the real-time dialogue method of digital human based on image cloning.

[0076] The wireless communication function of an electronic device can be realized through an antenna, a wireless communication module, a modem processor, and a baseband processor.

[0077] The wireless communication module can provide wireless communication solutions for electronic devices including wireless LAN, Bluetooth, global navigation satellite system, frequency modulation, short-range wireless communication technology, infrared technology, etc.

[0078] This embodiment also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for real-time conversation of digital humans based on image cloning are implemented.

[0079] A method for real-time conversation of digital human based on image cloning includes: obtaining original user video; dividing and arranging the original user video into closed-mouth silent video and non-silent video based on the mouth state of the user in the original user video; calculating the bounding box of the face area in the non-silent video based on the face detection algorithm, and recording the coordinate information of the bounding box; intercepting the image of the face area in the non-silent video frame by frame based on the bounding box; synthesizing the training video with the image of the face area; training the image cloning model based on the training video to obtain the trained image cloning model; obtaining the dialogue text input by the user in real time, and converting the dialogue text into the image cloning model; The text is input into the large language model to generate the answer text, and the speech synthesis model is used to convert the answer text into audio data; the audio data is imported into the trained image cloning model to generate a face image sequence that matches the audio data; the face image sequence is synthesized into the non-silent video according to the coordinate information of the bounding box to obtain the conversation video. When the duration of playing the face image sequence exceeds the duration of the non-silent video, the duration of the non-silent video is extended by using a forward and reverse loop playback method; the audio data and the conversation video are pushed to the front end synchronously; in the absence of conversation, the closed-mouth silent video is pushed to the front end in a loop.

[0080] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0081] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A digital human real-time dialogue method based on image cloning, characterized in that: The following steps are involved: Get the original user video; The original user video is segmented and sorted into closed-mouth silent videos and non-silent videos based on the user's mouth state in the original user video; Calculate the bounding box of the face area in the non-silent video based on the face detection algorithm and record the coordinate information of the bounding box; Capture images of face areas in non-silent videos frame by frame based on bounding boxes; Use images of face regions to synthesize training videos; The image cloning model is trained based on the training video to obtain a trained image cloning model; Acquire the conversation text input by the user in real time, input the conversation text into the large language model to generate the answer text, and use the speech synthesis model to convert the answer text into audio data; The audio data is imported into the trained image cloning model to generate a facial image sequence that matches the audio data; the facial image sequence is synthesized into the non-silent video according to the coordinate information of the bounding box to obtain a conversation video; when the duration of playing the facial image sequence exceeds the duration of the non-silent video, the duration of the non-silent video is extended by using a forward and reverse loop playback method; Push audio data and conversation video to the front end synchronously; In the absence of dialogue, push the silent video loop to the front end.

2. The method for real-time digital human dialogue based on image cloning according to claim 1 is characterized in that: Based on the face detection algorithm, the bounding box of the face area in the non-silent video is calculated, and the coordinate information of the bounding box is recorded, including: The position of the face area in each frame of the non-silent video is calculated based on the face detection algorithm, the position of the bounding box that can contain the face area in each frame is calculated based on the position of the face area in each frame, and the coordinate information of the bounding box is recorded based on the position of the bounding box.

3. The method for real-time digital human dialogue based on image cloning according to claim 1 is characterized in that: The image cloning model is an ER-NeRF model or a SyncTalk model.

4. The method for real-time digital human dialogue based on image cloning according to claim 1 is characterized in that: After the audio data is imported into the trained image cloning model and a facial image sequence matching the audio data is generated, the following steps are also included: Smooth the face image sequence to eliminate inter-frame jitter.

5. The method for real-time digital human dialogue based on image cloning according to claim 1 is characterized in that: The forward and reverse order loop playback method comprises the following steps: When the non-silent video is played to the end, the non-silent video is played in reverse. When the non-silent video is played in reverse to the starting position of the non-silent video, the non-silent video is played forward, and the cycle is repeated until the duration of the face image sequence is reached.

6. The method for real-time digital human dialogue based on image cloning according to claim 1 is characterized in that: Push audio data and conversation video to the front end synchronously; In the absence of dialogue, push the closed-mouth silent video loop to the front end including: Push audio data and conversation video to the front end synchronously through WebRTC technology; In the absence of dialogue, the silent video loop is pushed to the front end through WebRTC technology.

7. The method for real-time digital human dialogue based on image cloning according to claim 1 is characterized in that: The original user video contains a full-body or half-body picture, the user's movement range does not exceed the shoulder range, the original user video is not less than 5 minutes, and the length of the silent video with closed mouth is not less than 10 seconds.

8. A digital human real-time dialogue system based on image cloning, characterized in that: include: A video acquisition module, the video acquisition module is configured to obtain original user videos; A video processing module, the video processing module is configured to segment and organize the original user video into a closed-mouth silent video and a non-silent video based on the user's mouth state in the original user video; A face processing module, the face processing module is configured to calculate a bounding box of a face area in a non-silent video based on a face detection algorithm, record coordinate information of the bounding box; and capture an image of the face area in the non-silent video frame by frame based on the bounding box; Use images of face regions to synthesize training videos; A model training module, the model training module is configured to train the image cloning model based on the training video to obtain a trained image cloning model; A dialogue processing module, the dialogue processing module is configured to obtain dialogue text input by the user in real time, input the dialogue text into the large language model to generate answer text, and use the speech synthesis model to convert the answer text into audio data; The picture generation module is configured to import the audio data into the trained image cloning model to generate a facial image sequence matching the audio data; synthesize the facial image sequence into the non-silent video according to the coordinate information of the bounding box to obtain a conversation video, and when the duration of playing the facial image sequence exceeds the duration of the non-silent video, the duration of the non-silent video is extended by using a forward and reverse loop playback method; A real-time push module, the real-time push module is configured to push the audio data and the conversation video to the front end synchronously; In the absence of dialogue, push the silent video loop to the front end.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the digital human real-time dialogue method based on image cloning as described in any one of claims 1 to 7 are implemented.

10. A medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the digital human real-time dialogue method based on image cloning as described in any one of claims 1 to 7 are implemented.

Images